2024-09-13T13:40:54.740 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T13:40:54.758 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:40:54.838 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517 branch: wip-aclamk-cbt-combined description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_5925} email: akupczyk@redhat.com first_in_suite: false job_id: '7903517' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client ping interval: 4 mon client ping timeout: 12 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ mon_bind_msgr2: false sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 4159 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 682/13280 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKRmsnzUXjOwlKxFbs/p0+6SbpLAYzrKL97YBDV6soqI8S4JWLjmkXvFkT+4GF3T4ONlJIlFk+T6OHXrONTzdRI= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGAxtLVpH/50v2kZHnKQWFZcFqVHlmEJAmUCo0Recw5zl74xG/hPOpS2uzoqXwORqKJsVQLzTm+lZv98lyrXLKc= tasks: - install: null - ceph: null - mon_thrash: freeze_mon_duration: 20 freeze_mon_probability: 10 revive_delay: 20 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:43:51 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T13:40:54.838 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T13:40:54.839 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T13:40:54.839 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T13:40:54.841 INFO:teuthology.task.internal:Checking packages... 2024-09-13T13:40:54.861 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T13:40:54.861 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T13:40:54.861 INFO:teuthology.packaging:ref: None 2024-09-13T13:40:54.861 INFO:teuthology.packaging:tag: None 2024-09-13T13:40:54.861 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T13:40:54.861 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:40:54.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T13:40:55.017 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697.gd58f7306 2024-09-13T13:40:55.019 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T13:40:55.025 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T13:40:55.025 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T13:40:55.044 INFO:teuthology.task.internal:Saving configuration 2024-09-13T13:40:55.057 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T13:40:55.064 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T13:40:55.095 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:35:45.707041', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKRmsnzUXjOwlKxFbs/p0+6SbpLAYzrKL97YBDV6soqI8S4JWLjmkXvFkT+4GF3T4ONlJIlFk+T6OHXrONTzdRI='} 2024-09-13T13:40:55.127 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:35:45.705446', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGAxtLVpH/50v2kZHnKQWFZcFqVHlmEJAmUCo0Recw5zl74xG/hPOpS2uzoqXwORqKJsVQLzTm+lZv98lyrXLKc='} 2024-09-13T13:40:55.127 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T13:40:55.133 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-09-13T13:40:55.133 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-13T13:40:55.134 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T13:40:55.211 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f34f285fbe0>, signals=[15]) 2024-09-13T13:40:55.211 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T13:40:55.217 INFO:teuthology.task.internal:Opening connections... 2024-09-13T13:40:55.217 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:40:55.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:55.303 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:40:55.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:40:55.405 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T13:40:55.412 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-09-13T13:40:55.430 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-09-13T13:40:55.430 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-09-13T13:40:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-09-13T13:40:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-09-13T13:40:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-09-13T13:40:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:40:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:40:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:40:55.490 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-09-13T13:40:55.516 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-09-13T13:40:55.535 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-09-13T13:40:55.536 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-09-13T13:40:55.594 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-09-13T13:40:55.594 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T13:40:55.595 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T13:40:55.596 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-09-13T13:40:55.624 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T13:40:55.633 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T13:40:55.638 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T13:40:55.639 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:40:55.642 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:40:55.659 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T13:40:55.666 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T13:40:55.666 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:40:55.699 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:40:55.806 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T13:40:55.877 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T13:40:55.891 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T13:40:55.891 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T13:40:55.891 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:40:55.892 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:40:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:40:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:40:55.910 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-13T13:40:55.911 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-13T13:40:55.968 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:40:55.968 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-503.el9.x86_64 2024-09-13T13:40:55.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-09-13T13:40:55.969 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-503.el9.x86_64 2024-09-13T13:40:55.969 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-503.el9.x86_64 2024-09-13T13:40:55.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-09-13T13:40:58.636 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.2 MB 00:00 2024-09-13T13:40:59.550 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-13T13:41:00.903 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-13T13:41:01.834 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-13T13:41:05.246 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T13:41:06.243 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-13T13:41:07.014 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-13T13:41:07.810 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 91 kB/s | 19 kB 00:00 2024-09-13T13:41:08.442 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T13:41:09.248 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T13:41:13.881 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-13T13:41:14.075 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:01 ago on Fri 13 Sep 2024 01:41:13 PM UTC. 2024-09-13T13:41:14.628 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-09-13T13:41:15.879 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:41:15.880 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:41:15.964 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:41:15.974 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:41:15.974 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-13T13:41:15.974 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:41:15.974 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:41:15.974 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:41:15.974 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:41:15.975 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 79 M 2024-09-13T13:41:15.976 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 126 M 2024-09-13T13:41:15.976 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:41:16.307 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 80 kB/s | 23 kB 00:00 2024-09-13T13:41:16.642 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T13:41:16.643 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T13:41:16.727 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T13:41:16.738 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 79 M 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 126 M 2024-09-13T13:41:16.739 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:41:16.954 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 252 kB/s | 23 kB 00:00 2024-09-13T13:41:19.671 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 5.6 MB/s | 16 MB 00:02 2024-09-13T13:41:20.862 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 7.3 MB/s | 29 MB 00:03 2024-09-13T13:41:22.437 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-09-13T13:41:22.438 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:41:22.438 INFO:teuthology.orchestra.run.smithi121.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-13T13:41:23.129 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:41:23.449 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:41:23.449 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:41:24.408 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:41:24.409 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:41:26.868 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:41:27.553 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:41:27.654 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:41:30.438 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:41:30.575 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:41:32.197 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 991 kB/s | 16 MB 00:16 2024-09-13T13:41:35.414 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:41:35.492 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:41:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:41:45.546 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 1.0 MB/s | 29 MB 00:29 2024-09-13T13:41:56.345 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 877 kB/s | 35 MB 00:40 2024-09-13T13:41:56.345 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:41:56.345 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.0 MB/s | 79 MB 00:40 2024-09-13T13:41:57.027 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:41:57.401 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:41:57.401 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:41:58.335 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:41:58.336 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:42:00.798 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:42:01.495 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:42:01.588 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:42:04.411 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:42:04.541 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:42:09.395 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:42:09.565 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:14.411 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:18.658 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:18.698 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:18.987 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:42:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:42:18.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:42:19.814 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:42:20.155 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:42:20.239 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:42:20.239 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:42:20.240 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:42:20.240 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:42:20.240 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:42:20.240 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T13:42:20.240 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:42:20.241 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:42:20.298 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:42:20.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:42:20.299 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 82.el9 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8274670 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-09-13T13:42:20.331 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:42:20.332 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-09-13T13:42:20.333 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:42:20.333 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:42:20.334 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:42:20.334 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:42:20.334 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:42:20.948 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-09-13T13:42:22.273 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:42:22.305 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-09-13T13:42:22.307 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-13T13:42:22.336 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.w6HetPol4s 2024-09-13T13:42:22.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w6HetPol4s 2024-09-13T13:42:22.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.w6HetPol4s 2024-09-13T13:42:22.548 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.w6HetPol4s is 6KB 2024-09-13T13:42:22.597 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.w6HetPol4s 2024-09-13T13:42:22.611 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:42:22.674 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:42:22.674 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:42:22.675 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:42:22.675 INFO:teuthology.orchestra.run.smithi121.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:42:22.676 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:42:22.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-09-13T13:42:52.668 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:52.712 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:52.816 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T13:42:52.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-13T13:42:52.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:42:53.107 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:53.107 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T13:42:53.107 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T13:42:53.107 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:42:53.951 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:42:54.330 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:42:54.412 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:42:54.412 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:42:54.412 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T13:42:54.412 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:42:54.412 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:42:54.412 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-09-13T13:42:54.412 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:42:54.412 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T13:42:54.470 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T13:42:54.471 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T13:42:54.471 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 82.el9 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8274670 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-09-13T13:42:54.504 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T13:42:54.505 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-09-13T13:42:54.506 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:42:54.506 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T13:42:54.506 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T13:42:54.507 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-505.el9.x86_64 2024-09-13T13:42:54.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T13:42:55.162 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-09-13T13:42:56.498 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T13:42:56.527 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-09-13T13:42:56.533 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-13T13:42:56.549 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.lwJehhGKWO 2024-09-13T13:42:56.549 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lwJehhGKWO 2024-09-13T13:42:56.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.lwJehhGKWO 2024-09-13T13:42:56.768 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.lwJehhGKWO is 6KB 2024-09-13T13:42:56.817 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.lwJehhGKWO 2024-09-13T13:42:56.832 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T13:42:56.895 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T13:42:56.895 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T13:42:56.895 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T13:42:56.896 INFO:teuthology.orchestra.run.smithi153.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T13:42:56.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T13:42:57.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-09-13T13:43:27.092 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-09-13T13:43:27.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-13T13:43:27.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:43:45.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-13T13:43:52.836 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T13:43:54.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-13T13:43:54.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:43:57.533 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-13T13:44:01.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-13T13:44:01.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:44:02.240 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-13T13:44:02.636 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-13T13:44:02.636 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:44:02.636 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-09-13T13:44:02.651 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:44:02.651 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:44:02.651 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:44:02.651 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:44:03.652 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T13:44:03.652 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:44:03.739 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-09-13T13:44:03.758 DEBUG:teuthology.parallel:result is None 2024-09-13T13:44:09.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-13T13:44:09.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:44:41.729 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-13T13:44:42.140 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-13T13:44:42.140 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.14.0-505.el9.x86_64"... 2024-09-13T13:44:42.140 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-13T13:44:42.155 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-505.el9.x86_64 2024-09-13T13:44:42.155 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T13:44:42.155 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:44:42.155 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T13:44:43.156 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-09-13T13:44:43.156 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:44:43.247 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-09-13T13:44:43.270 DEBUG:teuthology.parallel:result is None 2024-09-13T13:44:43.270 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T13:44:43.279 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T13:44:43.280 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:44:43.283 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:44:43.299 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T13:44:43.305 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T13:44:43.314 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T13:44:43.315 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:44:43.342 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:44:43.371 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T13:44:43.377 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T13:44:43.378 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:44:43.409 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:44:43.441 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:44:43.453 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:44:43.472 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:44:43.482 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:44:43.484 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T13:44:43.491 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T13:44:43.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:44:43.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:44:43.522 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T13:44:43.531 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T13:44:43.531 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:44:43.565 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:44:43.579 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:44:43.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:44:43.707 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:44:43.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:44:43.829 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:44:43.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:44:43.894 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:44:43.942 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:44:43.983 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:44:44.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:44:44.107 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:44:44.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:44:44.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-09-13T13:44:44.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-09-13T13:44:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:44:44.256 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:44:44.633 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T13:44:44.643 INFO:teuthology.task.internal:Starting timer... 2024-09-13T13:44:44.643 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T13:44:44.653 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T13:44:44.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-09-13T13:44:44.712 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-09-13T13:44:44.713 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-09-13T13:44:44.768 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-09-13T13:44:44.770 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T13:44:44.770 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-09-13T13:44:44.809 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-09-13T13:44:44.810 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-09-13T13:44:44.832 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-09-13T13:44:44.832 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-09-13T13:44:44.833 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:44:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:44:44.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:44:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:44:44.902 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T13:44:44.902 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:44:44.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-09-13T13:44:44.985 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T13:44:44.997 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T13:44:45.008 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T13:44:45.009 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T13:48:27.148 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-09-13T13:48:27.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-13T13:48:27.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:48:27.239 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-09-13T13:48:27.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-09-13T13:48:27.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-13T13:48:27.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:48:27.403 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-13T13:48:27.480 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-13T13:48:27.480 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T13:48:27.490 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T13:48:27.491 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:48:27.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:48:27.493 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:48:27.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:48:27.521 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T13:48:27.536 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T13:48:27.550 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T13:48:27.561 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-09-13T13:48:27.564 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T13:48:27.572 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-09-13T13:48:27.587 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T13:48:27.588 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-09-13T13:48:27.599 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-09-13T13:48:27.600 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T13:48:27.614 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T13:48:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T13:48:27.652 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:48:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:48:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-13T13:48:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.655 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.656 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.690 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:48:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:48:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-13T13:48:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.693 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:48:27.694 INFO:teuthology.run_tasks:Running task install... 2024-09-13T13:48:27.702 DEBUG:teuthology.task.install:project ceph 2024-09-13T13:48:27.703 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T13:48:27.703 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T13:48:27.703 INFO:teuthology.task.install:Using flavor: default 2024-09-13T13:48:27.708 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T13:48:27.708 INFO:teuthology.task.install:extra packages: [] 2024-09-13T13:48:27.712 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T13:48:27.713 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:48:27.715 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'tag': None, 'wait_for_package': False} 2024-09-13T13:48:27.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:48:27.868 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T13:48:27.868 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T13:48:27.883 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/ 2024-09-13T13:48:27.884 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4697.gd58f7306 2024-09-13T13:48:27.995 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T13:48:27.995 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:48:27.995 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T13:48:28.002 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T13:48:28.002 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:48:28.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T13:48:28.022 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T13:48:28.022 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T13:48:28.029 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T13:48:28.029 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d58f73069a8157a9ace12a0247ee3e8497707ea9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T13:48:28.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T13:48:28.094 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T13:48:28.162 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T13:48:28.169 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T13:48:28.226 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-09-13T13:48:28.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-09-13T13:48:28.234 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-09-13T13:48:28.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-09-13T13:48:28.457 INFO:teuthology.orchestra.run.smithi153.stdout:45 files removed 2024-09-13T13:48:28.462 INFO:teuthology.orchestra.run.smithi121.stdout:45 files removed 2024-09-13T13:48:28.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-09-13T13:48:28.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-09-13T13:48:29.038 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 314 kB/s | 84 kB 00:00 2024-09-13T13:48:29.043 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 306 kB/s | 84 kB 00:00 2024-09-13T13:48:29.321 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-13T13:48:29.327 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-13T13:48:29.597 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-13T13:48:29.619 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-13T13:48:31.414 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.2 MB 00:01 2024-09-13T13:48:33.611 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.2 MB 00:03 2024-09-13T13:48:35.471 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-13T13:48:39.914 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 20 MB 00:07 2024-09-13T13:48:40.213 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.5 MB 00:00 2024-09-13T13:48:41.780 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-13T13:48:43.059 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-13T13:48:44.389 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.5 MB 00:00 2024-09-13T13:48:46.306 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 19 kB 00:00 2024-09-13T13:48:47.769 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T13:48:48.457 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-13T13:48:50.842 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:48:50.849 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:48:50.849 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:48:50.849 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:48:50.849 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T13:48:50.850 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T13:48:50.851 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T13:48:50.852 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout:Install 30 Packages 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-09-13T13:48:50.853 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:48:50.854 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 62 M 2024-09-13T13:48:50.854 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:48:51.866 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 9.3 MB/s | 5.5 MB 00:00 2024-09-13T13:48:51.917 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 503 kB/s | 25 kB 00:00 2024-09-13T13:48:52.008 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-13T13:48:52.151 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 13 MB/s | 11 MB 00:00 2024-09-13T13:48:52.184 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 2.8 MB/s | 508 kB 00:00 2024-09-13T13:48:52.243 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 774 kB/s | 45 kB 00:00 2024-09-13T13:48:52.468 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): ceph-common-19.3.0-4697.gd58f7306.el9.x 18 MB/s | 22 MB 00:01 2024-09-13T13:48:52.502 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): python3-ceph-common-19.3.0-4697.gd58f73 553 kB/s | 143 kB 00:00 2024-09-13T13:48:52.527 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 2.7 MB/s | 161 kB 00:00 2024-09-13T13:48:52.552 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.2 MB/s | 321 kB 00:00 2024-09-13T13:48:52.577 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.9 MB/s | 305 kB 00:00 2024-09-13T13:48:52.603 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T13:48:52.686 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_ 10 MB/s | 5.5 MB 00:00 2024-09-13T13:48:52.770 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 556 kB/s | 107 kB 00:00 2024-09-13T13:48:52.820 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-09-13T13:48:52.846 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 160 kB 00:00 2024-09-13T13:48:52.896 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-13T13:48:52.921 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-13T13:48:52.946 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-13T13:48:52.972 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-13T13:48:53.005 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 808 kB/s | 47 kB 00:00 2024-09-13T13:48:53.031 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-13T13:48:53.081 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T13:48:53.106 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-13T13:48:53.148 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2024-09-13T13:48:53.207 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 154 kB/s | 26 kB 00:00 2024-09-13T13:48:53.224 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T13:48:53.257 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-13T13:48:53.324 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-13T13:48:53.349 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): re2-20211101-20.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-13T13:48:53.424 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-09-13T13:48:53.741 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 8.3 MB/s | 3.5 MB 00:00 2024-09-13T13:48:53.824 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 6.8 MB/s | 3.2 MB 00:00 2024-09-13T13:48:53.829 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:53.829 INFO:teuthology.orchestra.run.smithi153.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-13T13:48:53.925 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:48:53.960 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:48:53.961 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:48:54.698 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:48:54.698 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:48:55.584 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 11 M 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.5 M 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:48:55.596 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T13:48:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:48:55.597 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 22 M 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 25 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 755 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T13:48:55.600 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 508 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 5.5 M 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 45 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 143 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 321 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 305 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 99 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-13T13:48:55.601 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout:Install 30 Packages 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-09-13T13:48:55.602 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:48:55.603 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 62 M 2024-09-13T13:48:55.603 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:48:56.171 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:48:56.251 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T13:48:56.289 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T13:48:56.313 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T13:48:56.317 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-base-19.3.0-4697.gd58f7306.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-13T13:48:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:48:56.356 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:48:56.375 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-selinux-19.3.0-4697.gd58f7306.el9. 432 kB/s | 25 kB 00:00 2024-09-13T13:48:56.485 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): libcephfs2-19.3.0-4697.gd58f7306.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-13T13:48:56.546 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:48:56.599 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:48:56.635 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): ceph-radosgw-19.3.0-4697.gd58f7306.el9. 15 MB/s | 11 MB 00:00 2024-09-13T13:48:56.660 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): libradosstriper1-19.3.0-4697.gd58f7306. 2.8 MB/s | 508 kB 00:00 2024-09-13T13:48:56.727 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): python3-ceph-argparse-19.3.0-4697.gd58f 676 kB/s | 45 kB 00:00 2024-09-13T13:48:56.810 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-ceph-common-19.3.0-4697.gd58f73 1.7 MB/s | 143 kB 00:00 2024-09-13T13:48:56.886 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): librgw2-19.3.0-4697.gd58f7306.el9.x86_6 22 MB/s | 5.5 MB 00:00 2024-09-13T13:48:56.911 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-cephfs-19.3.0-4697.gd58f7306.el 1.6 MB/s | 161 kB 00:00 2024-09-13T13:48:56.936 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-rados-19.3.0-4697.gd58f7306.el 6.3 MB/s | 321 kB 00:00 2024-09-13T13:48:56.970 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-rbd-19.3.0-4697.gd58f7306.el9. 5.1 MB/s | 305 kB 00:00 2024-09-13T13:48:56.988 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T13:48:56.995 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-rgw-19.3.0-4697.gd58f7306.el9. 1.7 MB/s | 99 kB 00:00 2024-09-13T13:48:57.040 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:48:57.133 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:48:57.222 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T13:48:57.245 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): ceph-common-19.3.0-4697.gd58f7306.el9. 16 MB/s | 22 MB 00:01 2024-09-13T13:48:57.267 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:48:57.270 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-09-13T13:48:57.332 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:48:57.354 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 446 kB/s | 160 kB 00:00 2024-09-13T13:48:57.380 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-09-13T13:48:57.383 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T13:48:57.427 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T13:48:57.463 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-09-13T13:48:57.488 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 389 kB/s | 42 kB 00:00 2024-09-13T13:48:57.513 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:48:57.522 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-09-13T13:48:57.535 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:48:57.547 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 611 kB/s | 36 kB 00:00 2024-09-13T13:48:57.573 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-13T13:48:57.590 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-09-13T13:48:57.600 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:48:57.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T13:48:57.656 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-13T13:48:57.673 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 379 kB/s | 26 kB 00:00 2024-09-13T13:48:57.690 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-13T13:48:57.716 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-13T13:48:57.758 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-13T13:48:57.770 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T13:48:57.783 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.8 MB/s | 191 kB 00:00 2024-09-13T13:48:57.810 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T13:48:57.851 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T13:48:57.858 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-13T13:48:57.892 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T13:48:57.921 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T13:48:57.958 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T13:48:57.984 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-13T13:48:58.051 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): librados2-19.3.0-4697.gd58f7306.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-13T13:48:58.126 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): librbd1-19.3.0-4697.gd58f7306.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-13T13:48:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:48:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-13T13:48:58.232 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:48:58.268 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:48:58.268 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:48:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T13:48:58.325 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:48:58.413 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:48:58.454 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T13:48:58.499 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T13:48:58.961 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:48:58.962 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:48:58.994 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T13:48:59.027 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:48:59.099 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:48:59.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T13:48:59.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T13:49:00.443 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:49:00.527 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-13T13:49:00.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-13T13:49:00.581 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-13T13:49:00.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:49:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-13T13:49:00.805 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:49:00.859 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-13T13:49:01.261 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-13T13:49:01.325 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:49:01.417 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:49:01.508 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 8/34 2024-09-13T13:49:01.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:49:01.577 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:49:01.618 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:49:01.622 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 9/34 2024-09-13T13:49:01.660 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-13T13:49:01.704 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 11/34 2024-09-13T13:49:01.774 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:49:01.804 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:49:01.867 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 13/34 2024-09-13T13:49:01.902 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-13T13:49:02.020 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-13T13:49:02.053 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-13T13:49:02.094 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-13T13:49:02.126 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-13T13:49:02.155 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-13T13:49:02.205 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-13T13:49:02.223 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:49:02.271 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:49:02.362 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:49:02.363 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T13:49:02.363 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T13:49:02.363 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:02.397 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:49:02.530 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-13T13:49:02.576 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:49:02.666 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 22/34 2024-09-13T13:49:02.706 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 23/34 2024-09-13T13:49:02.741 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-13T13:49:03.232 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-13T13:49:03.270 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:49:03.341 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 26/34 2024-09-13T13:49:03.410 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 27/34 2024-09-13T13:49:03.439 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 28/34 2024-09-13T13:49:05.721 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:49:05.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:49:06.440 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 29/34 2024-09-13T13:49:06.472 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:49:06.557 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 30/34 2024-09-13T13:49:06.557 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T13:49:06.557 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T13:49:06.557 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:06.590 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:49:12.598 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-09-13T13:49:12.599 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:13.622 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:49:13.647 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:49:13.647 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:13.647 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T13:49:13.647 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:49:13.647 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:49:13.647 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:13.649 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:49:13.732 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:49:13.732 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T13:49:14.826 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T13:49:14.827 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T13:49:14.828 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T13:49:14.829 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:49:14.829 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T13:49:14.829 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T13:49:15.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.817 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T13:49:15.818 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T13:49:15.819 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T13:49:15.820 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:15.820 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:16.178 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-09-13T13:49:16.748 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:16.761 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:49:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-09-13T13:49:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-09-13T13:49:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-09-13T13:49:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-09-13T13:49:16.761 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-09-13T13:49:16.762 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-09-13T13:49:16.762 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-09-13T13:49:16.762 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:16.881 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:16.881 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:16.881 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:49:16.881 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:16.881 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-09-13T13:49:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:16.883 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 51 M 2024-09-13T13:49:16.883 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 216 M 2024-09-13T13:49:16.883 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:49:17.404 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-13T13:49:17.438 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 710 kB/s | 161 kB 00:00 2024-09-13T13:49:17.455 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T13:49:17.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:49:17.786 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 32/34 2024-09-13T13:49:17.786 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:17.786 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T13:49:17.786 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:49:17.786 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:49:17.786 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:17.787 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:49:17.862 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-13T13:49:17.862 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:49:18.764 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-13T13:49:18.766 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:18.766 INFO:teuthology.orchestra.run.smithi153.stdout:Total 27 MB/s | 51 MB 00:01 2024-09-13T13:49:18.794 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:49:18.814 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:49:18.814 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:49:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:49:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 1/34 2024-09-13T13:49:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2/34 2024-09-13T13:49:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 3/34 2024-09-13T13:49:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 4/34 2024-09-13T13:49:19.036 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 5/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_6 6/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 7/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9. 8/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x8 9/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 10/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 11/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 12/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 13/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-13T13:49:19.037 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-13T13:49:19.038 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 31/34 2024-09-13T13:49:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-13T13:49:19.039 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 33/34 2024-09-13T13:49:19.194 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:49:19.195 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:49:19.732 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:49:19.836 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T13:49:19.883 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T13:49:19.907 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T13:49:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T13:49:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T13:49:20.123 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:20.123 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:20.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-09-13T13:49:21.038 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:28 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:21.169 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 50 M 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:49:21.170 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 161 k 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-09-13T13:49:21.171 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:21.172 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 51 M 2024-09-13T13:49:21.172 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 216 M 2024-09-13T13:49:21.172 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:49:21.525 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): libcephsqlite-19.3.0-4697.gd58f7306.el9. 749 kB/s | 161 kB 00:00 2024-09-13T13:49:21.901 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-09-13T13:49:21.951 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 475 kB/s | 304 kB 00:00 2024-09-13T13:49:22.843 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): ceph-test-19.3.0-4697.gd58f7306.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-13T13:49:22.845 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:22.845 INFO:teuthology.orchestra.run.smithi121.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-13T13:49:22.873 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:49:22.893 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:49:22.893 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:49:23.276 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:49:23.276 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:49:23.808 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:49:23.899 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T13:49:23.949 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T13:49:23.986 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:49:24.673 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:49:24.747 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:49:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:49:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T13:49:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T13:49:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T13:49:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T13:49:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:27.462 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:49:27.462 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T13:49:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T13:49:27.463 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:27.463 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:27.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-09-13T13:49:28.351 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:40 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:28.495 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:28.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:28.498 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:49:28.498 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:28.498 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:49:28.498 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T13:49:28.499 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:49:28.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T13:49:28.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T13:49:28.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T13:49:28.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T13:49:28.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T13:49:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T13:49:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T13:49:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T13:49:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T13:49:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T13:49:28.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T13:49:28.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T13:49:28.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T13:49:28.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T13:49:28.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T13:49:28.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections 2024-09-13T13:49:28.502 INFO:teuthology.orchestra.run.smithi153.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T13:49:28.502 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T13:49:28.502 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T13:49:28.502 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T13:49:28.502 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T13:49:28.503 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T13:49:28.504 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T13:49:28.504 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T13:49:28.504 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T13:49:28.504 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T13:49:28.504 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T13:49:28.504 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T13:49:28.505 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T13:49:28.505 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T13:49:28.505 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T13:49:28.505 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:28.505 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:49:28.505 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:28.506 INFO:teuthology.orchestra.run.smithi153.stdout:Install 37 Packages 2024-09-13T13:49:28.506 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:28.507 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 31 M 2024-09-13T13:49:28.507 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 109 M 2024-09-13T13:49:28.507 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:49:28.770 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 3/4 2024-09-13T13:49:28.851 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:49:29.327 INFO:teuthology.orchestra.run.smithi153.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-13T13:49:29.485 INFO:teuthology.orchestra.run.smithi153.stdout:(2/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 5.2 MB/s | 1.5 MB 00:00 2024-09-13T13:49:29.561 INFO:teuthology.orchestra.run.smithi153.stdout:(3/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-13T13:49:29.619 INFO:teuthology.orchestra.run.smithi153.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 4.9 MB/s | 291 kB 00:00 2024-09-13T13:49:29.795 INFO:teuthology.orchestra.run.smithi153.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-13T13:49:29.895 INFO:teuthology.orchestra.run.smithi153.stdout:(6/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 8.4 MB/s | 4.8 MB 00:00 2024-09-13T13:49:30.070 INFO:teuthology.orchestra.run.smithi153.stdout:(7/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-13T13:49:30.104 INFO:teuthology.orchestra.run.smithi153.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2024-09-13T13:49:30.154 INFO:teuthology.orchestra.run.smithi153.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.5 MB/s | 125 kB 00:00 2024-09-13T13:49:30.204 INFO:teuthology.orchestra.run.smithi153.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 345 kB/s | 106 kB 00:00 2024-09-13T13:49:30.230 INFO:teuthology.orchestra.run.smithi153.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 850 kB/s | 135 kB 00:00 2024-09-13T13:49:30.255 INFO:teuthology.orchestra.run.smithi153.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-09-13T13:49:30.363 INFO:teuthology.orchestra.run.smithi153.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-13T13:49:30.397 INFO:teuthology.orchestra.run.smithi153.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 980 kB/s | 188 kB 00:00 2024-09-13T13:49:30.422 INFO:teuthology.orchestra.run.smithi153.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 896 kB/s | 172 kB 00:00 2024-09-13T13:49:30.448 INFO:teuthology.orchestra.run.smithi153.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 920 kB/s | 77 kB 00:00 2024-09-13T13:49:30.464 INFO:teuthology.orchestra.run.smithi153.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 759 kB/s | 51 kB 00:00 2024-09-13T13:49:30.481 INFO:teuthology.orchestra.run.smithi153.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 707 kB/s | 42 kB 00:00 2024-09-13T13:49:30.541 INFO:teuthology.orchestra.run.smithi153.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-13T13:49:30.557 INFO:teuthology.orchestra.run.smithi153.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 204 kB/s | 22 kB 00:00 2024-09-13T13:49:30.574 INFO:teuthology.orchestra.run.smithi153.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-13T13:49:30.608 INFO:teuthology.orchestra.run.smithi153.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.6 MB/s | 172 kB 00:00 2024-09-13T13:49:30.624 INFO:teuthology.orchestra.run.smithi153.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 247 kB/s | 11 kB 00:00 2024-09-13T13:49:30.641 INFO:teuthology.orchestra.run.smithi153.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 556 kB/s | 18 kB 00:00 2024-09-13T13:49:30.658 INFO:teuthology.orchestra.run.smithi153.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 692 kB/s | 23 kB 00:00 2024-09-13T13:49:30.675 INFO:teuthology.orchestra.run.smithi153.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-13T13:49:30.692 INFO:teuthology.orchestra.run.smithi153.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 585 kB/s | 20 kB 00:00 2024-09-13T13:49:30.709 INFO:teuthology.orchestra.run.smithi153.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-13T13:49:30.734 INFO:teuthology.orchestra.run.smithi153.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-13T13:49:30.768 INFO:teuthology.orchestra.run.smithi153.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-09-13T13:49:30.802 INFO:teuthology.orchestra.run.smithi153.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-13T13:49:30.818 INFO:teuthology.orchestra.run.smithi153.stdout:(32/37): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-09-13T13:49:30.844 INFO:teuthology.orchestra.run.smithi153.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-13T13:49:30.861 INFO:teuthology.orchestra.run.smithi153.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-13T13:49:30.877 INFO:teuthology.orchestra.run.smithi153.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-13T13:49:30.903 INFO:teuthology.orchestra.run.smithi153.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-13T13:49:30.928 INFO:teuthology.orchestra.run.smithi153.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-13T13:49:30.932 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:30.932 INFO:teuthology.orchestra.run.smithi153.stdout:Total 13 MB/s | 31 MB 00:02 2024-09-13T13:49:31.083 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:49:31.123 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:49:31.124 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:49:31.217 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 4/4 2024-09-13T13:49:31.217 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 1/4 2024-09-13T13:49:31.217 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2/4 2024-09-13T13:49:31.217 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T13:49:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T13:49:31.556 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:31.556 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:49:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T13:49:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T13:49:31.557 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:31.557 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:31.602 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:49:31.602 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:49:31.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-09-13T13:49:32.440 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:39 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:32.587 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 6.6 k 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:49:32.589 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 2.4 M 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 1.5 M 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 291 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 4.8 M 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 17 M 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T13:49:32.590 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T13:49:32.591 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout:Install 37 Packages 2024-09-13T13:49:32.592 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:32.593 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 M 2024-09-13T13:49:32.593 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 109 M 2024-09-13T13:49:32.593 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:49:32.971 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:49:33.031 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T13:49:33.072 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T13:49:33.100 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T13:49:33.172 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T13:49:33.224 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T13:49:33.256 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T13:49:33.347 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T13:49:33.427 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T13:49:33.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T13:49:33.511 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T13:49:33.541 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T13:49:33.575 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T13:49:33.608 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T13:49:33.649 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T13:49:33.692 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T13:49:33.694 INFO:teuthology.orchestra.run.smithi121.stdout:(1/37): ceph-19.3.0-4697.gd58f7306.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-13T13:49:33.726 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T13:49:33.775 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T13:49:33.824 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T13:49:33.896 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T13:49:33.928 INFO:teuthology.orchestra.run.smithi121.stdout:(2/37): ceph-mds-19.3.0-4697.gd58f7306.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-13T13:49:33.961 INFO:teuthology.orchestra.run.smithi121.stdout:(3/37): ceph-mgr-19.3.0-4697.gd58f7306.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-09-13T13:49:34.044 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T13:49:34.045 INFO:teuthology.orchestra.run.smithi121.stdout:(4/37): ceph-mon-19.3.0-4697.gd58f7306.el9.x86_ 14 MB/s | 4.8 MB 00:00 2024-09-13T13:49:34.070 INFO:teuthology.orchestra.run.smithi121.stdout:(5/37): ceph-mgr-modules-core-19.3.0-4697.gd58f 2.6 MB/s | 291 kB 00:00 2024-09-13T13:49:34.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T13:49:34.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T13:49:34.224 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T13:49:34.245 INFO:teuthology.orchestra.run.smithi121.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-13T13:49:34.273 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T13:49:34.304 INFO:teuthology.orchestra.run.smithi121.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-13T13:49:34.329 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T13:49:34.488 INFO:teuthology.orchestra.run.smithi121.stdout:(8/37): ceph-osd-19.3.0-4697.gd58f7306.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-13T13:49:34.521 INFO:teuthology.orchestra.run.smithi121.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 2.7 MB/s | 1.2 MB 00:00 2024-09-13T13:49:34.528 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T13:49:34.547 INFO:teuthology.orchestra.run.smithi121.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 558 kB/s | 135 kB 00:00 2024-09-13T13:49:34.572 INFO:teuthology.orchestra.run.smithi121.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-09-13T13:49:34.578 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T13:49:34.656 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T13:49:34.706 INFO:teuthology.orchestra.run.smithi121.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-13T13:49:34.731 INFO:teuthology.orchestra.run.smithi121.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-13T13:49:34.743 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T13:49:34.756 INFO:teuthology.orchestra.run.smithi121.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 692 kB/s | 35 kB 00:00 2024-09-13T13:49:34.781 INFO:teuthology.orchestra.run.smithi121.stdout:(15/37): python3-urllib3-1.26.5-6.el9.noarch.rp 827 kB/s | 215 kB 00:00 2024-09-13T13:49:34.786 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T13:49:34.807 INFO:teuthology.orchestra.run.smithi121.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-13T13:49:34.832 INFO:teuthology.orchestra.run.smithi121.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-13T13:49:34.857 INFO:teuthology.orchestra.run.smithi121.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-13T13:49:34.915 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T13:49:34.916 INFO:teuthology.orchestra.run.smithi121.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 749 kB/s | 43 kB 00:00 2024-09-13T13:49:34.941 INFO:teuthology.orchestra.run.smithi121.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-13T13:49:34.966 INFO:teuthology.orchestra.run.smithi121.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-09-13T13:49:35.008 INFO:teuthology.orchestra.run.smithi121.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-13T13:49:35.037 INFO:teuthology.orchestra.run.smithi121.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 669 kB/s | 18 kB 00:00 2024-09-13T13:49:35.062 INFO:teuthology.orchestra.run.smithi121.stdout:(24/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 112 kB/s | 11 kB 00:00 2024-09-13T13:49:35.075 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T13:49:35.079 INFO:teuthology.orchestra.run.smithi121.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 556 kB/s | 23 kB 00:00 2024-09-13T13:49:35.096 INFO:teuthology.orchestra.run.smithi121.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 575 kB/s | 19 kB 00:00 2024-09-13T13:49:35.104 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:49:35.113 INFO:teuthology.orchestra.run.smithi121.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 581 kB/s | 20 kB 00:00 2024-09-13T13:49:35.130 INFO:teuthology.orchestra.run.smithi121.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-13T13:49:35.130 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:49:35.130 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:35.130 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T13:49:35.131 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:49:35.131 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:49:35.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:35.155 INFO:teuthology.orchestra.run.smithi121.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-09-13T13:49:35.181 INFO:teuthology.orchestra.run.smithi121.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-13T13:49:35.206 INFO:teuthology.orchestra.run.smithi121.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-13T13:49:35.231 INFO:teuthology.orchestra.run.smithi121.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-13T13:49:35.256 INFO:teuthology.orchestra.run.smithi121.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-13T13:49:35.290 INFO:teuthology.orchestra.run.smithi121.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T13:49:35.315 INFO:teuthology.orchestra.run.smithi121.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-13T13:49:35.340 INFO:teuthology.orchestra.run.smithi121.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-13T13:49:35.374 INFO:teuthology.orchestra.run.smithi121.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-13T13:49:35.379 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:35.379 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-13T13:49:35.527 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:49:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:49:35.567 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:49:36.047 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:49:36.047 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:49:36.613 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:49:36.641 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:49:36.641 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:36.641 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T13:49:36.641 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:49:36.641 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:49:36.641 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:37.089 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:49:37.113 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:49:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T13:49:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:49:37.113 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:49:37.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:37.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:49:37.363 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:49:37.363 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:37.363 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T13:49:37.363 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:49:37.364 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:49:37.364 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:37.396 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:49:37.427 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:49:37.460 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-13T13:49:37.501 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-13T13:49:37.529 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-13T13:49:37.601 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-13T13:49:37.662 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-13T13:49:37.695 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-13T13:49:37.788 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-13T13:49:37.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-13T13:49:37.915 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-13T13:49:37.958 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-13T13:49:37.988 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-13T13:49:38.022 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-13T13:49:38.055 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-13T13:49:38.104 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-13T13:49:38.139 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-13T13:49:38.174 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-13T13:49:38.223 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-13T13:49:38.254 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-13T13:49:38.332 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-13T13:49:38.484 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-13T13:49:38.565 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-13T13:49:38.627 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-13T13:49:38.665 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-13T13:49:38.714 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-13T13:49:38.769 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-13T13:49:38.957 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-13T13:49:39.011 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-13T13:49:39.089 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-13T13:49:39.160 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:49:39.160 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T13:49:39.160 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T13:49:39.161 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T13:49:39.162 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T13:49:39.163 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T13:49:39.163 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T13:49:39.164 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T13:49:39.164 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T13:49:39.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T13:49:39.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T13:49:39.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T13:49:39.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T13:49:39.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T13:49:39.165 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T13:49:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-13T13:49:39.218 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-13T13:49:39.348 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-13T13:49:39.509 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 32/37 2024-09-13T13:49:39.554 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:49:39.580 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 33/37 2024-09-13T13:49:39.580 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:39.580 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T13:49:39.580 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:49:39.581 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T13:49:39.581 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:39.917 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T13:49:39.917 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:39.917 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:49:39.917 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:39.917 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T13:49:39.918 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T13:49:39.919 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:39.920 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:40.329 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-09-13T13:49:40.904 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:40.997 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:49:41.040 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:41.041 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:49:41.041 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:49:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 34/37 2024-09-13T13:49:41.091 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:41.091 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T13:49:41.091 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:49:41.091 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T13:49:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:41.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-09-13T13:49:41.564 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:49:41.588 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 35/37 2024-09-13T13:49:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T13:49:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:49:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T13:49:41.588 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:41.679 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:41.812 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:49:41.814 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:41.815 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:49:41.815 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:41.815 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 783 k 2024-09-13T13:49:41.815 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 789 k 2024-09-13T13:49:41.815 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:49:41.815 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:49:41.839 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 36/37 2024-09-13T13:49:41.839 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:41.839 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T13:49:41.839 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:49:41.839 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T13:49:41.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:41.904 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:49:42.097 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 2.7 MB/s | 783 kB 00:00 2024-09-13T13:49:42.098 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:42.098 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.7 MB/s | 783 kB 00:00 2024-09-13T13:49:42.098 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:49:42.103 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:49:42.103 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:49:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:49:42.110 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:49:42.191 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:49:42.651 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:42.760 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:43.120 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:43.395 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:43.395 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:43.396 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:49:43.396 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:43.396 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:43.396 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:43.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T13:49:43.922 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 37/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 1/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 3/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 4/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 5/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9. 6/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-13T13:49:43.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-13T13:49:43.924 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-13T13:49:43.925 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-13T13:49:44.188 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:44.323 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:44.324 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 145 k 2024-09-13T13:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 432 k 2024-09-13T13:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:49:44.529 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 713 kB/s | 145 kB 00:00 2024-09-13T13:49:44.529 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:44.530 INFO:teuthology.orchestra.run.smithi153.stdout:Total 706 kB/s | 145 kB 00:00 2024-09-13T13:49:44.530 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:49:44.537 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:49:44.537 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:49:44.598 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:49:44.598 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:49:44.751 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:49:44.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:49:44.863 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:49:44.863 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:44.863 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T13:49:44.864 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-13T13:49:44.950 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:44.950 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T13:49:44.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T13:49:44.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T13:49:44.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T13:49:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T13:49:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T13:49:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T13:49:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T13:49:44.954 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:44.954 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:45.318 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-09-13T13:49:45.772 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:49:45.772 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:45.772 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:49:45.772 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:45.772 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:45.772 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:45.886 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:52 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:45.976 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:49:46.015 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-09-13T13:49:46.018 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:46.020 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:49:46.020 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:46.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-09-13T13:49:46.596 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:46.657 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:46.689 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:49:46.731 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:46.732 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:49:46.732 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:46.791 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:46.791 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:49:46.792 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:46.792 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:49:46.792 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 783 k 2024-09-13T13:49:46.793 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:46.793 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:49:46.793 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:46.793 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:49:46.794 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:46.794 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 783 k 2024-09-13T13:49:46.794 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 789 k 2024-09-13T13:49:46.794 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:49:46.813 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T13:49:47.050 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.3.0-4697.gd58f7306.el9.noarch.rpm 3.0 MB/s | 783 kB 00:00 2024-09-13T13:49:47.050 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:47.050 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.9 MB/s | 783 kB 00:00 2024-09-13T13:49:47.051 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:49:47.055 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:49:47.056 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:49:47.061 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:49:47.062 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:49:47.131 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:49:47.374 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:47.506 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:47.507 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T13:49:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-09-13T13:49:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:47.510 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 19 M 2024-09-13T13:49:47.510 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 159 M 2024-09-13T13:49:47.510 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:49:47.595 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:47.678 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:48.015 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 232 kB/s | 34 kB 00:00 2024-09-13T13:49:48.037 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:48.040 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 95 kB/s | 16 kB 00:00 2024-09-13T13:49:48.233 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-13T13:49:48.272 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:49:48.272 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:48.272 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:49:48.272 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:48.272 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:48.272 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:48.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T13:49:48.525 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 22 MB/s | 14 MB 00:00 2024-09-13T13:49:48.616 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.9 MB/s | 548 kB 00:00 2024-09-13T13:49:48.641 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 782 kB/s | 19 kB 00:00 2024-09-13T13:49:48.675 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.5 MB/s | 1.0 MB 00:00 2024-09-13T13:49:48.742 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-13T13:49:48.767 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-13T13:49:48.809 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-13T13:49:48.901 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.8 MB/s | 2.0 MB 00:00 2024-09-13T13:49:49.048 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:56 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:49.185 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:49.185 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 145 k 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:49:49.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:49.187 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-09-13T13:49:49.187 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 432 k 2024-09-13T13:49:49.187 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:49:49.318 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 795 kB/s | 863 kB 00:01 2024-09-13T13:49:49.322 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:49.322 INFO:teuthology.orchestra.run.smithi153.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-13T13:49:49.371 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:49:49.390 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.3.0-4697.gd58f73 715 kB/s | 145 kB 00:00 2024-09-13T13:49:49.390 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:49:49.390 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:49:49.391 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:49.391 INFO:teuthology.orchestra.run.smithi121.stdout:Total 708 kB/s | 145 kB 00:00 2024-09-13T13:49:49.391 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:49:49.398 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:49:49.398 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:49:49.455 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:49:49.455 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:49:49.609 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:49:49.664 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:49:49.664 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:49:49.707 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:49:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:49:49.732 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:49:49.732 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T13:49:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:50.258 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:49:50.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T13:49:50.379 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T13:49:50.426 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T13:49:50.456 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T13:49:50.574 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T13:49:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306. 1/1 2024-09-13T13:49:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:50.671 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:49:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:49:50.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:50.672 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:50.699 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T13:49:50.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-09-13T13:49:50.969 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T13:49:51.020 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T13:49:51.048 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:49:51.080 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T13:49:51.508 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:58 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:51.596 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:49:51.638 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:49:51.639 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:49:51.716 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T13:49:52.279 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:49:52.411 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:49:52.412 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 14 M 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 34 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 16 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-13T13:49:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 19 M 2024-09-13T13:49:52.414 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 159 M 2024-09-13T13:49:52.415 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:49:53.010 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4697.gd5 242 kB/s | 34 kB 00:00 2024-09-13T13:49:53.036 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4697.gd58 99 kB/s | 16 kB 00:00 2024-09-13T13:49:53.528 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4697.gd58f730 22 MB/s | 14 MB 00:00 2024-09-13T13:49:53.669 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 425 kB/s | 269 kB 00:00 2024-09-13T13:49:53.736 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-13T13:49:53.764 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 773 kB/s | 19 kB 00:00 2024-09-13T13:49:53.787 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-09-13T13:49:53.837 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-13T13:49:53.871 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-09-13T13:49:53.896 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 527 kB/s | 31 kB 00:00 2024-09-13T13:49:53.955 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-13T13:49:53.997 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.7 MB/s | 2.0 MB 00:00 2024-09-13T13:49:54.000 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:49:54.000 INFO:teuthology.orchestra.run.smithi121.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-13T13:49:54.047 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:49:54.066 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:49:54.066 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:49:54.264 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T13:49:54.334 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:49:54.350 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:49:54.351 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:49:54.945 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:49:55.031 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-13T13:49:55.066 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-13T13:49:55.121 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-13T13:49:55.151 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-13T13:49:55.258 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-13T13:49:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-13T13:49:55.478 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-13T13:49:55.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-13T13:49:55.551 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:49:55.584 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 10/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T13:49:56.203 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T13:49:56.204 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T13:49:56.204 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T13:49:56.204 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:49:56.204 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T13:49:56.204 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T13:49:56.572 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T13:49:56.572 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T13:49:56.573 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T13:49:56.574 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:56.574 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:49:56.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T13:49:57.386 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:09 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:49:57.525 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T13:49:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-09-13T13:49:57.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:49:57.528 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 38 M 2024-09-13T13:49:57.528 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 212 M 2024-09-13T13:49:57.528 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:49:57.733 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-13T13:49:57.759 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.8 MB/s | 801 kB 00:00 2024-09-13T13:49:57.809 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-09-13T13:49:57.842 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 549 kB/s | 18 kB 00:00 2024-09-13T13:49:57.888 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 848 kB/s | 38 kB 00:00 2024-09-13T13:49:58.151 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-46 13 MB/s | 7.4 MB 00:00 2024-09-13T13:49:58.277 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-13T13:49:58.535 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-09-13T13:49:58.627 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-13T13:49:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 11/12 2024-09-13T13:49:58.821 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:49:59.128 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.0 MB/s | 4.9 MB 00:01 2024-09-13T13:49:59.619 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-13T13:50:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 12/12 2024-09-13T13:50:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el 1/12 2024-09-13T13:50:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noa 2/12 2024-09-13T13:50:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9 3/12 2024-09-13T13:50:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-13T13:50:00.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-13T13:50:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-13T13:50:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-13T13:50:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-13T13:50:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-13T13:50:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-13T13:50:00.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-13T13:50:01.027 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-13T13:50:01.028 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-13T13:50:01.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T13:50:01.029 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T13:50:01.029 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:01.029 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:01.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T13:50:01.867 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:08 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:02.000 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:02.001 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-09-13T13:50:02.001 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 7.4 M 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T13:50:02.002 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-09-13T13:50:02.003 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:02.004 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 38 M 2024-09-13T13:50:02.004 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 212 M 2024-09-13T13:50:02.004 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:02.470 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 847 kB/s | 191 kB 00:00 2024-09-13T13:50:02.537 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-09-13T13:50:02.573 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-13T13:50:02.646 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 243 kB/s | 18 kB 00:00 2024-09-13T13:50:02.738 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-46 15 MB/s | 7.4 MB 00:00 2024-09-13T13:50:02.755 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 354 kB/s | 38 kB 00:00 2024-09-13T13:50:02.805 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-13T13:50:02.872 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-13T13:50:02.886 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.8 MB/s | 16 MB 00:04 2024-09-13T13:50:02.887 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:02.887 INFO:teuthology.orchestra.run.smithi153.stdout:Total 7.1 MB/s | 38 MB 00:05 2024-09-13T13:50:02.947 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-09-13T13:50:02.973 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-13T13:50:03.243 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:03.269 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:03.269 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:03.273 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-09-13T13:50:03.506 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 6.4 MB/s | 4.9 MB 00:00 2024-09-13T13:50:03.507 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:03.507 INFO:teuthology.orchestra.run.smithi121.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-13T13:50:03.704 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:03.704 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:03.864 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:03.892 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:03.893 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:04.336 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:04.336 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:04.472 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:04.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T13:50:04.592 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:50:04.691 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:50:04.735 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T13:50:04.921 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T13:50:04.950 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T13:50:05.041 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T13:50:05.100 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:05.174 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T13:50:05.213 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:50:05.321 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:50:05.364 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T13:50:05.547 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T13:50:05.588 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T13:50:05.670 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T13:50:05.716 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T13:50:05.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T13:50:06.339 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T13:50:06.475 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T13:50:07.075 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:50:07.707 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:50:08.104 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T13:50:08.141 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:50:08.754 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T13:50:08.788 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T13:50:10.698 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T13:50:10.699 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:50:10.699 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T13:50:11.075 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:11.076 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 12/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7 1/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T13:50:11.436 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T13:50:11.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T13:50:11.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T13:50:11.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T13:50:11.437 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T13:50:11.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-09-13T13:50:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T13:50:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:11.838 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:11.838 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:11.839 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:12.040 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:12.174 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T13:50:12.175 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-09-13T13:50:12.176 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:12.177 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.2 M 2024-09-13T13:50:12.177 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-09-13T13:50:12.177 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:12.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-09-13T13:50:12.649 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 222 kB/s | 19 kB 00:00 2024-09-13T13:50:12.674 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 229 kB/s | 26 kB 00:00 2024-09-13T13:50:12.733 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 289 kB/s | 49 kB 00:00 2024-09-13T13:50:12.758 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-13T13:50:12.792 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-13T13:50:12.799 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:19 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:12.817 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-13T13:50:12.859 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-13T13:50:12.885 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-13T13:50:12.910 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-13T13:50:12.932 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 49 k 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T13:50:12.933 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T13:50:12.934 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:12.935 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:12.935 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:12.935 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-09-13T13:50:12.935 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:12.935 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.2 M 2024-09-13T13:50:12.936 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-09-13T13:50:12.936 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:12.943 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-13T13:50:12.977 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.4 MB/s | 90 kB 00:00 2024-09-13T13:50:13.020 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-13T13:50:13.044 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-rsa-4.9-2.el9.noarch.rpm 441 kB/s | 59 kB 00:00 2024-09-13T13:50:13.047 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:13.047 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-13T13:50:13.111 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:13.126 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:13.126 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:13.139 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 19 kB 00:00 2024-09-13T13:50:13.165 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 139 kB/s | 26 kB 00:00 2024-09-13T13:50:13.190 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): ceph-mgr-rook-19.3.0-4697.gd58f7306.el9 235 kB/s | 49 kB 00:00 2024-09-13T13:50:13.214 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:13.215 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:13.240 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-13T13:50:13.265 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-13T13:50:13.291 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-13T13:50:13.333 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 581 kB/s | 54 kB 00:00 2024-09-13T13:50:13.358 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 348 kB/s | 32 kB 00:00 2024-09-13T13:50:13.383 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 165 kB/s | 14 kB 00:00 2024-09-13T13:50:13.417 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.8 MB/s | 59 kB 00:00 2024-09-13T13:50:13.442 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-13T13:50:13.467 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.6 MB/s | 220 kB 00:00 2024-09-13T13:50:13.501 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.5 MB/s | 1.0 MB 00:00 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.9 MB/s | 2.2 MB 00:00 2024-09-13T13:50:13.568 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:13.583 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:13.583 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:13.670 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:13.672 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:13.725 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:13.900 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T13:50:13.961 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T13:50:14.025 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T13:50:14.063 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T13:50:14.104 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T13:50:14.175 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:14.215 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T13:50:14.317 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T13:50:14.348 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T13:50:14.373 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T13:50:14.402 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T13:50:14.458 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T13:50:14.503 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T13:50:14.545 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T13:50:14.647 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T13:50:14.748 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T13:50:14.797 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T13:50:15.101 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T13:50:15.153 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T13:50:15.187 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T13:50:15.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T13:50:15.264 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:50:15.517 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T13:50:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T13:50:15.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T13:50:15.651 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T13:50:15.672 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:50:16.091 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:50:16.091 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T13:50:16.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T13:50:16.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T13:50:16.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T13:50:16.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T13:50:16.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T13:50:16.092 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T13:50:16.093 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T13:50:16.093 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T13:50:16.093 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T13:50:16.093 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T13:50:16.093 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 13/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 1/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T13:50:16.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T13:50:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T13:50:16.407 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T13:50:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T13:50:16.413 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:16.413 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T13:50:16.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:16.415 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:16.650 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T13:50:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T13:50:16.732 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T13:50:16.733 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T13:50:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T13:50:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T13:50:16.734 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:16.734 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:16.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T13:50:17.214 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:17.349 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T13:50:17.350 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-09-13T13:50:17.351 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:17.352 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 7.0 M 2024-09-13T13:50:17.352 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-09-13T13:50:17.352 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:17.560 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:24 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:17.679 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.3 MB/s | 248 kB 00:00 2024-09-13T13:50:17.694 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:17.695 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:17.695 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 154 k 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T13:50:17.696 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:17.697 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:17.697 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:17.697 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-09-13T13:50:17.697 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:17.697 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 7.0 M 2024-09-13T13:50:17.697 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-09-13T13:50:17.698 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:17.729 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-13T13:50:17.755 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-13T13:50:17.780 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-13T13:50:17.805 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 666 kB/s | 154 kB 00:00 2024-09-13T13:50:17.947 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:17.949 INFO:teuthology.orchestra.run.smithi153.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-13T13:50:18.020 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:18.030 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:18.131 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:18.131 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:18.321 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4697.gd58f7306.e 715 kB/s | 154 kB 00:00 2024-09-13T13:50:18.387 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-09-13T13:50:18.413 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-13T13:50:18.430 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 4.4 MB/s | 74 kB 00:00 2024-09-13T13:50:18.447 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:18.615 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T13:50:18.666 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T13:50:18.938 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 298 kB/s | 248 kB 00:00 2024-09-13T13:50:19.196 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T13:50:19.286 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T13:50:19.297 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 6.0 MB 00:01 2024-09-13T13:50:19.299 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:19.300 INFO:teuthology.orchestra.run.smithi121.stdout:Total 4.4 MB/s | 7.0 MB 00:01 2024-09-13T13:50:19.363 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T13:50:19.373 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:19.385 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:19.386 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:50:19.486 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:19.487 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:19.805 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:19.984 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T13:50:20.032 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T13:50:20.151 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:50:20.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T13:50:20.151 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T13:50:20.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T13:50:20.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T13:50:20.152 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T13:50:20.444 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T13:50:20.444 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:20.445 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:20.578 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T13:50:20.668 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T13:50:20.703 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-09-13T13:50:20.736 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T13:50:20.761 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:50:21.272 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:21.407 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:21.408 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-09-13T13:50:21.409 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:21.409 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 979 k 2024-09-13T13:50:21.409 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.9 M 2024-09-13T13:50:21.409 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:21.510 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 6/6 2024-09-13T13:50:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 1/6 2024-09-13T13:50:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T13:50:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T13:50:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T13:50:21.511 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T13:50:21.541 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 722 kB/s | 80 kB 00:00 2024-09-13T13:50:21.741 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 2.8 MB/s | 900 kB 00:00 2024-09-13T13:50:21.741 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:21.742 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.9 MB/s | 979 kB 00:00 2024-09-13T13:50:21.754 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:21.776 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:21.776 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:21.806 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:21.807 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:21.857 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:21.858 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:22.040 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:22.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-09-13T13:50:22.186 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T13:50:22.258 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:50:22.634 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:22.767 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 900 k 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:22.768 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:22.769 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-09-13T13:50:22.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:22.769 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 979 k 2024-09-13T13:50:22.769 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.9 M 2024-09-13T13:50:22.769 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:22.907 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-09-13T13:50:22.972 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:50:22.972 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T13:50:23.057 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.3.0-4697.gd58f7306.el9.x86_ 3.3 MB/s | 900 kB 00:00 2024-09-13T13:50:23.057 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:23.058 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.3 MB/s | 979 kB 00:00 2024-09-13T13:50:23.069 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:23.091 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:23.091 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:23.171 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:23.172 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:23.247 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T13:50:23.247 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:23.248 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:23.248 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T13:50:23.248 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:23.248 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:23.362 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:23.495 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-09-13T13:50:23.501 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T13:50:23.564 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:50:24.081 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:24.222 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 285 k 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.3 M 2024-09-13T13:50:24.223 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2/2 2024-09-13T13:50:24.260 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/2 2024-09-13T13:50:24.454 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T13:50:24.454 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:24.455 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T13:50:24.455 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:24.463 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:24.463 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:24.497 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:24.497 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:24.515 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T13:50:24.516 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:24.516 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:24.516 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T13:50:24.516 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:24.516 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:24.716 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:24.781 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:50:24.797 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:50:24.797 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:50:24.797 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T13:50:24.797 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:24.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-09-13T13:50:25.374 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:25.509 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:25.510 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-09-13T13:50:25.510 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:25.510 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.3.0-4697.gd58f7306.el9 ceph-noarch 285 k 2024-09-13T13:50:25.510 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 285 k 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.3 M 2024-09-13T13:50:25.511 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:25.745 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-volume-19.3.0-4697.gd58f7306.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-13T13:50:25.746 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:25.746 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-13T13:50:25.746 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:25.754 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:25.754 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:25.784 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:25.784 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:25.832 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:50:25.832 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:25.832 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:25.833 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:25.833 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:25.833 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:26.003 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:26.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:50:26.079 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:50:26.079 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:50:26.079 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T13:50:26.080 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:26.134 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-09-13T13:50:26.718 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:26.856 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:26.857 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:26.858 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:26.858 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:50:26.858 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:26.858 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-09-13T13:50:26.858 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-09-13T13:50:26.858 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:27.068 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 605 kB/s | 127 kB 00:00 2024-09-13T13:50:27.068 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:27.068 INFO:teuthology.orchestra.run.smithi153.stdout:Total 600 kB/s | 127 kB 00:00 2024-09-13T13:50:27.068 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:27.074 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:27.074 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:27.113 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 1/1 2024-09-13T13:50:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:27.114 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.3.0-4697.gd58f7306.el9.noarch 2024-09-13T13:50:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:27.114 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:27.139 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:27.139 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:27.288 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:27.382 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:27.416 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-09-13T13:50:27.898 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:27.998 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:28.123 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:28.124 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:28.124 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:28.124 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:28.124 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:28.124 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:28.136 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:28.136 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 127 k 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-09-13T13:50:28.137 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:28.350 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.3.0-4697.gd58f7306.el9.x86_64 595 kB/s | 127 kB 00:00 2024-09-13T13:50:28.351 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:28.351 INFO:teuthology.orchestra.run.smithi121.stdout:Total 591 kB/s | 127 kB 00:00 2024-09-13T13:50:28.351 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:28.357 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:28.357 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:28.398 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-09-13T13:50:28.421 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:28.422 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:28.568 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:28.656 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:28.989 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:29.080 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:29.125 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:29.126 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:29.126 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:29.143 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:29.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-09-13T13:50:29.389 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:29.390 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:29.390 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:29.390 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:29.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-09-13T13:50:29.768 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:29.902 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:29.902 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:29.902 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 32 k 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 142 k 2024-09-13T13:50:29.903 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:30.054 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 210 kB/s | 32 kB 00:00 2024-09-13T13:50:30.055 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:30.055 INFO:teuthology.orchestra.run.smithi153.stdout:Total 208 kB/s | 32 kB 00:00 2024-09-13T13:50:30.055 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:30.059 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:30.059 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:30.089 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:30.089 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:30.177 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:30.212 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:30.293 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:30.305 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:30.351 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:30.352 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:30.352 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:30.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-09-13T13:50:30.684 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:30.908 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:30.908 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:30.908 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:30.908 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:30.908 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:30.908 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:30.982 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:37 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:31.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-09-13T13:50:31.115 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:31.115 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:31.115 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-13T13:50:31.115 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 32 k 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 32 k 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 142 k 2024-09-13T13:50:31.116 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:31.271 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.3.0-4697.gd58f7306.el9.x86_6 205 kB/s | 32 kB 00:00 2024-09-13T13:50:31.272 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:31.272 INFO:teuthology.orchestra.run.smithi121.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-13T13:50:31.272 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:31.275 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:31.275 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:31.295 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:31.296 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:31.384 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:31.501 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:31.654 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:31.746 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:31.789 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:31.790 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:31.791 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:31.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-09-13T13:50:31.864 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:32.105 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:32.105 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:32.105 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:32.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-09-13T13:50:32.429 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:32.519 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:32.562 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:32.563 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:32.563 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:32.636 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-09-13T13:50:32.858 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:32.948 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:32.990 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:32.991 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:32.991 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:33.063 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-09-13T13:50:33.203 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:33.293 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:33.337 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:33.338 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:33.338 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:33.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-09-13T13:50:33.620 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:33.709 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:33.752 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:33.752 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:33.753 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:33.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-09-13T13:50:33.980 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:34.071 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:34.114 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:34.115 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:34.115 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:34.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-09-13T13:50:34.394 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:34.485 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:34.528 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:34.529 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:34.529 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:34.599 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-09-13T13:50:34.751 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:34.842 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:34.885 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:34.886 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:34.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-09-13T13:50:35.165 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:35.254 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:35.297 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:35.298 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:35.298 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:35.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-09-13T13:50:35.518 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:47 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:35.609 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:35.652 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:35.653 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-09-13T13:50:35.653 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:35.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-09-13T13:50:35.938 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:36.028 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:36.072 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:36.073 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:36.073 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:36.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-09-13T13:50:36.281 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:48 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:36.413 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:36.413 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:50:36.414 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:36.415 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2024-09-13T13:50:36.415 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 231 k 2024-09-13T13:50:36.415 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:36.608 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 443 kB/s | 85 kB 00:00 2024-09-13T13:50:36.608 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:36.609 INFO:teuthology.orchestra.run.smithi153.stdout:Total 439 kB/s | 85 kB 00:00 2024-09-13T13:50:36.609 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:36.616 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:36.616 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:36.679 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:36.680 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:36.700 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:36.788 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.3.0-4697.gd58f7306.el9.x86_64 is already installed. 2024-09-13T13:50:36.822 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:36.832 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:36.833 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-09-13T13:50:36.833 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:36.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-09-13T13:50:36.922 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:37.315 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:37.475 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:44 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:37.582 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:37.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:37.583 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:37.583 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:37.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:37.583 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:37.608 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:37.609 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:37.609 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-13T13:50:37.609 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:37.609 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:37.609 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 85 k 2024-09-13T13:50:37.609 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-09-13T13:50:37.610 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:37.799 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.3.0-4697.gd58f7306.el9.x86_64.rpm 452 kB/s | 85 kB 00:00 2024-09-13T13:50:37.799 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:37.799 INFO:teuthology.orchestra.run.smithi121.stdout:Total 449 kB/s | 85 kB 00:00 2024-09-13T13:50:37.799 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:37.806 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:37.807 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:37.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-09-13T13:50:37.868 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:37.869 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:38.012 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:38.104 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:38.470 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:50 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:38.495 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:38.609 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:38.610 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.2 M 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 14 M 2024-09-13T13:50:38.611 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:38.739 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:38.739 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:38.740 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:38.740 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:38.740 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:38.740 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:38.944 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 9.5 MB/s | 3.2 MB 00:00 2024-09-13T13:50:38.945 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:38.945 INFO:teuthology.orchestra.run.smithi153.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-13T13:50:38.946 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:38.955 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:38.955 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:39.008 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-09-13T13:50:39.042 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:39.042 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:39.500 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:39.570 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T13:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:50:39.597 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:39.612 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:46 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:39.758 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 3.2 M 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:39.759 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.2 M 2024-09-13T13:50:39.760 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 14 M 2024-09-13T13:50:39.760 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:40.078 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.3.0-4697.gd58f7306.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-13T13:50:40.079 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:40.079 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.9 MB/s | 3.2 MB 00:00 2024-09-13T13:50:40.079 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:40.088 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:40.088 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:40.179 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:40.179 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:40.734 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:40.734 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:40.734 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:40.734 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:40.734 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:40.735 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:40.738 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T13:50:40.738 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T13:50:40.738 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:50:40.738 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T13:50:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:40.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-09-13T13:50:41.578 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:53 ago on Fri 13 Sep 2024 01:48:48 PM UTC. 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-09-13T13:50:41.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:41.718 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 170 k 2024-09-13T13:50:41.718 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 490 k 2024-09-13T13:50:41.718 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-09-13T13:50:41.828 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:41.829 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 825 kB/s | 170 kB 00:00 2024-09-13T13:50:41.924 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi153.stdout:Total 820 kB/s | 170 kB 00:00 2024-09-13T13:50:41.925 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-09-13T13:50:41.932 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-09-13T13:50:41.933 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-09-13T13:50:41.998 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-09-13T13:50:41.998 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-09-13T13:50:42.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-09-13T13:50:42.149 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-09-13T13:50:42.256 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:42.640 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 01:48:53 PM UTC. 2024-09-13T13:50:42.658 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:42.779 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.3.0-4697.gd58f7306.el9 ceph 170 k 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-09-13T13:50:42.780 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-09-13T13:50:42.781 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-09-13T13:50:42.781 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:42.781 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 170 k 2024-09-13T13:50:42.781 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 490 k 2024-09-13T13:50:42.781 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-09-13T13:50:42.886 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:42.886 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:42.886 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-09-13T13:50:42.886 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:42.886 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:42.886 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-09-13T13:50:42.999 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.3.0-4697.gd58f7306.el9.x86_64.rpm 782 kB/s | 170 kB 00:00 2024-09-13T13:50:42.999 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-09-13T13:50:42.999 INFO:teuthology.orchestra.run.smithi121.stdout:Total 777 kB/s | 170 kB 00:00 2024-09-13T13:50:43.000 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-09-13T13:50:43.007 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-09-13T13:50:43.007 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-09-13T13:50:43.071 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-09-13T13:50:43.071 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-09-13T13:50:43.129 DEBUG:teuthology.parallel:result is None 2024-09-13T13:50:43.219 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-09-13T13:50:43.320 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:43.740 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:43.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 1/1 2024-09-13T13:50:43.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:43.999 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-09-13T13:50:43.999 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.3.0-4697.gd58f7306.el9.x86_64 2024-09-13T13:50:43.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:43.999 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-09-13T13:50:44.238 DEBUG:teuthology.parallel:result is None 2024-09-13T13:50:44.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:50:44.389 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T13:50:44.410 INFO:teuthology.orchestra.run.smithi121.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T13:50:44.411 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T13:50:44.411 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T13:50:44.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:50:44.563 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T13:50:44.588 INFO:teuthology.orchestra.run.smithi153.stdout:19.3.0-4697.gd58f7306.el9 2024-09-13T13:50:44.589 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697.gd58f7306.el9 2024-09-13T13:50:44.589 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697.gd58f7306 is installed. 2024-09-13T13:50:44.591 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T13:50:44.591 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:44.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:50:44.616 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:44.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:50:44.658 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T13:50:44.658 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:44.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:50:44.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:50:44.748 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:44.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:50:44.777 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:50:44.845 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T13:50:44.845 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:44.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:50:44.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:50:44.944 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:44.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:50:44.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:50:45.041 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T13:50:45.041 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:45.042 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:50:45.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:50:45.131 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:45.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:50:45.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:50:45.228 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T13:50:45.322 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T13:50:45.322 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-09-13T13:50:45.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-09-13T13:50:45.352 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T13:50:45.352 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T13:50:45.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T13:50:45.418 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T13:50:45.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T13:50:45.456 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T13:50:45.495 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T13:50:45.495 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-13T13:50:45.496 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517', 'branch': 'wip-aclamk-cbt-combined', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_5925}', 'email': 'akupczyk@redhat.com', 'first_in_suite': False, 'job_id': '7903517', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-aclamk-cbt-combined'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_'], 'mon_bind_msgr2': False, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-aclamk-cbt-combined', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}}, 'owner': 'scheduled_akupczyk@teuthology', 'priority': 160, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 4159, 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'sleep_before_teardown': 0, 'subset': '682/13280', 'suite': 'rados', 'suite_branch': 'wip-aclamk-cbt-combined', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9', 'targets': {'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKRmsnzUXjOwlKxFbs/p0+6SbpLAYzrKL97YBDV6soqI8S4JWLjmkXvFkT+4GF3T4ONlJIlFk+T6OHXrONTzdRI=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGAxtLVpH/50v2kZHnKQWFZcFqVHlmEJAmUCo0Recw5zl74xG/hPOpS2uzoqXwORqKJsVQLzTm+lZv98lyrXLKc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'freeze_mon_duration': 20, 'freeze_mon_probability': 10, 'revive_delay': 20, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_09:43:51', 'tube': 'smithi', 'user': 'akupczyk', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T13:50:45.496 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T13:50:45.521 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T13:50:45.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T13:50:45.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T13:50:45.605 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:45.606 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T13:50:45.655 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T13:50:45.656 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T13:50:45.711 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-13 13:50:41.011680946 +0000 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-13 13:48:07.084266547 +0000 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-13 13:48:07.084266547 +0000 2024-09-13T13:50:45.712 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-13 13:48:07.084266547 +0000 2024-09-13T13:50:45.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T13:50:45.776 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-13T13:50:45.777 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-13T13:50:45.777 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000197002 s, 2.6 MB/s 2024-09-13T13:50:45.777 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T13:50:45.833 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-09-13T13:50:45.888 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T13:50:45.888 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:45.888 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-13T13:50:45.888 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:45.888 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:45.888 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-13 13:50:41.011680946 +0000 2024-09-13T13:50:45.889 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-13 13:48:07.421262135 +0000 2024-09-13T13:50:45.889 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-13 13:48:07.421262135 +0000 2024-09-13T13:50:45.889 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-13 13:48:07.421262135 +0000 2024-09-13T13:50:45.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T13:50:45.950 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-13T13:50:45.950 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-13T13:50:45.950 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000221106 s, 2.3 MB/s 2024-09-13T13:50:45.952 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T13:50:46.008 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-13 13:50:41.011680946 +0000 2024-09-13T13:50:46.064 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-13 13:48:07.745257890 +0000 2024-09-13T13:50:46.065 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-13 13:48:07.745257890 +0000 2024-09-13T13:50:46.065 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-13 13:48:07.745257890 +0000 2024-09-13T13:50:46.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-13T13:50:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000220212 s, 2.3 MB/s 2024-09-13T13:50:46.129 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T13:50:46.186 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-09-13 13:50:41.012680933 +0000 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-09-13 13:48:08.082253474 +0000 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-09-13 13:48:08.082253474 +0000 2024-09-13T13:50:46.241 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-09-13 13:48:08.082253474 +0000 2024-09-13T13:50:46.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T13:50:46.305 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-09-13T13:50:46.305 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-09-13T13:50:46.305 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00021574 s, 2.4 MB/s 2024-09-13T13:50:46.306 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T13:50:46.362 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T13:50:46.362 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:46.363 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T13:50:46.382 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T13:50:46.382 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-09-13T13:50:46.439 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-13 13:50:39.847411598 +0000 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-13 13:48:08.397625070 +0000 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-13 13:48:08.397625070 +0000 2024-09-13T13:50:46.440 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-13 13:48:08.397625070 +0000 2024-09-13T13:50:46.440 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T13:50:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-13T13:50:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-13T13:50:46.506 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000276104 s, 1.9 MB/s 2024-09-13T13:50:46.508 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T13:50:46.565 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-09-13T13:50:46.622 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T13:50:46.622 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:46.622 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-13T13:50:46.622 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:46.622 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:46.622 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-13 13:50:39.847411598 +0000 2024-09-13T13:50:46.623 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-13 13:48:08.719619453 +0000 2024-09-13T13:50:46.623 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-13 13:48:08.719619453 +0000 2024-09-13T13:50:46.623 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-13 13:48:08.719619453 +0000 2024-09-13T13:50:46.623 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T13:50:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-13T13:50:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-13T13:50:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000217883 s, 2.3 MB/s 2024-09-13T13:50:46.689 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T13:50:46.745 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-13 13:50:39.847411598 +0000 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-13 13:48:09.043613802 +0000 2024-09-13T13:50:46.802 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-13 13:48:09.043613802 +0000 2024-09-13T13:50:46.803 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-13 13:48:09.043613802 +0000 2024-09-13T13:50:46.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T13:50:46.865 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-13T13:50:46.866 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-13T13:50:46.866 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000172438 s, 3.0 MB/s 2024-09-13T13:50:46.867 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T13:50:46.923 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-13 13:50:39.847411598 +0000 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-13 13:48:09.374608029 +0000 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-13 13:48:09.374608029 +0000 2024-09-13T13:50:46.980 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-09-13 13:48:09.374608029 +0000 2024-09-13T13:50:46.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T13:50:47.043 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-13T13:50:47.043 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-13T13:50:47.044 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000165734 s, 3.1 MB/s 2024-09-13T13:50:47.045 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T13:50:47.102 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T13:50:47.102 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T13:50:47.102 INFO:tasks.ceph:Generating config... 2024-09-13T13:50:47.103 INFO:tasks.ceph:[client] debug monc = 20 2024-09-13T13:50:47.103 INFO:tasks.ceph:[client] debug ms = 1 2024-09-13T13:50:47.103 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-13T13:50:47.103 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-13T13:50:47.103 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-13T13:50:47.103 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-13T13:50:47.104 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-13T13:50:47.104 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-13T13:50:47.104 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-13T13:50:47.104 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-13T13:50:47.104 INFO:tasks.ceph:[global] ms type = async 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-13T13:50:47.104 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-13T13:50:47.105 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-13T13:50:47.105 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-13T13:50:47.105 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-13T13:50:47.105 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-13T13:50:47.105 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-13T13:50:47.105 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T13:50:47.105 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] mon client ping interval = 4 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] mon client ping timeout = 12 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-13T13:50:47.106 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T13:50:47.107 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-13T13:50:47.107 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-13T13:50:47.107 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T13:50:47.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T13:50:47.157 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T13:50:47.160 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T13:50:47.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:50:47.229 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.121:6789]'), ('mon.b', '[v1:172.21.15.121:6790]'), ('mon.c', '[v1:172.21.15.121:6791]'), ('mon.d', '[v1:172.21.15.121:6792]'), ('mon.e', '[v1:172.21.15.121:6793]'), ('mon.f', '[v1:172.21.15.153:6789]'), ('mon.g', '[v1:172.21.15.153:6790]'), ('mon.h', '[v1:172.21.15.153:6791]'), ('mon.i', '[v1:172.21.15.153:6792]')] 2024-09-13T13:50:47.229 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.121:6789],[v1:172.21.15.121:6790],[v1:172.21.15.121:6791],[v1:172.21.15.121:6792],[v1:172.21.15.121:6793],[v1:172.21.15.153:6789],[v1:172.21.15.153:6790],[v1:172.21.15.153:6791],[v1:172.21.15.153:6792]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client ping interval': 4, 'mon client ping timeout': 12, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-09-13T13:50:47.230 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:47.230 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T13:50:47.285 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.121:6789]' --addv b '[v1:172.21.15.121:6790]' --addv c '[v1:172.21.15.121:6791]' --addv d '[v1:172.21.15.121:6792]' --addv e '[v1:172.21.15.121:6793]' --addv f '[v1:172.21.15.153:6789]' --addv g '[v1:172.21.15.153:6790]' --addv h '[v1:172.21.15.153:6791]' --addv i '[v1:172.21.15.153:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:47.391 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: generated fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:setting min_mon_release = quincy 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 0 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:min_mon_release 17 (quincy) 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:election_strategy: 3 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:1: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:2: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:3: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:50:47.392 INFO:teuthology.orchestra.run.smithi121.stdout:4: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:50:47.393 INFO:teuthology.orchestra.run.smithi121.stdout:5: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:50:47.393 INFO:teuthology.orchestra.run.smithi121.stdout:6: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:50:47.393 INFO:teuthology.orchestra.run.smithi121.stdout:7: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:50:47.393 INFO:teuthology.orchestra.run.smithi121.stdout:8: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:50:47.393 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-09-13T13:50:47.395 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T13:50:47.412 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 21ebbdf6-8985-4682-9234-cdd2d2c1c2db... 2024-09-13T13:50:47.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T13:50:47.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T13:50:47.495 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-09-13T13:50:47.495 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-09-13T13:50:47.495 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T13:50:47.495 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-09-13T13:50:47.495 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T13:50:47.496 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-09-13T13:50:47.497 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-09-13T13:50:47.498 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "[v1:172.21.15.121:6789],[v1:172.21.15.121:6790],[v1:172.21.15.121:6791],[v1:172.21.15.121:6792],[v1:172.21.15.121:6793],[v1:172.21.15.153:6789],[v1:172.21.15.153:6790],[v1:172.21.15.153:6791],[v1:172.21.15.153:6792]" 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: mon client directed command retry = 5 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: mon election default strategy = 3 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms bind msgr2 = False 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject delay max = 1 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject delay probability = 0.005 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject delay type = mon 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject internal delays = 0.002 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms inject socket failures = 2500 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: ms type = async 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-09-13T13:50:47.499 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.500 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-09-13T13:50:47.501 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression algorithm = zlib 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore compression mode = aggressive 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore write v2 = False 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore zero block detection = True 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: mon client ping interval = 4 2024-09-13T13:50:47.502 INFO:teuthology.orchestra.run.smithi121.stdout: mon client ping timeout = 12 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify cached snaps = True 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify missing on start = True 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock override recovery settings = True 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue = debug_random 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: osd op queue cut off = debug_random 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-09-13T13:50:47.503 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 10 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-09-13T13:50:47.504 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T13:50:47.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon min osdmap epochs = 25 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune interval = 2 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune min = 15 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon osdmap full prune txsize = 2 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub inject crc mismatch = 0.01 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub inject missing keys = 0.05 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: mon scrub interval = 300 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: paxos service trim min = 5 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:50:47.506 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: debug monc = 20 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.d] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.e] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.f] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.g] 2024-09-13T13:50:47.507 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.h] 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.i] 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.508 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.509 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T13:50:47.510 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "[v1:172.21.15.121:6789],[v1:172.21.15.121:6790],[v1:172.21.15.121:6791],[v1:172.21.15.121:6792],[v1:172.21.15.121:6793],[v1:172.21.15.153:6789],[v1:172.21.15.153:6790],[v1:172.21.15.153:6791],[v1:172.21.15.153:6792]" 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-09-13T13:50:47.511 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 3 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr2 = False 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay max = 1 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay probability = 0.005 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay type = mon 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject internal delays = 0.002 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 2500 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-09-13T13:50:47.512 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-09-13T13:50:47.513 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression algorithm = zlib 2024-09-13T13:50:47.514 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression mode = aggressive 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore write v2 = False 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: mon client ping interval = 4 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: mon client ping timeout = 12 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-09-13T13:50:47.515 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:50:47.516 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 10 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-09-13T13:50:47.517 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-09-13T13:50:47.518 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 25 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject crc mismatch = 0.01 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject missing keys = 0.05 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 5 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T13:50:47.519 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.d] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.e] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.f] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.g] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.h] 2024-09-13T13:50:47.520 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.i] 2024-09-13T13:50:47.522 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T13:50:47.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T13:50:47.616 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T13:50:47.617 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:47.617 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T13:50:47.632 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:47.632 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T13:50:47.687 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:50:47.688 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:47.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T13:50:47.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:50:47.761 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:50:47.761 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:47.817 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:50:47.817 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:47.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T13:50:47.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T13:50:47.855 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:50:47.855 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:47.911 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T13:50:47.911 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T13:50:47.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T13:50:48.045 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T13:50:48.048 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T13:50:48.048 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T13:50:48.049 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T13:50:48.089 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T13:50:48.102 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T13:50:48.103 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-13T13:50:48.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T13:50:48.128 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T13:50:48.128 INFO:tasks.ceph:role: osd.0 2024-09-13T13:50:48.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:50:48.129 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T13:50:48.201 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:50:48.201 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:50:48.201 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:50:48.201 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:50:48.201 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:50:48.201 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:50:48.202 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:50:48.202 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:50:48.202 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:50:48.202 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:50:48.710 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-13T13:50:48.712 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-13T13:50:48.712 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T13:50:48.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T13:50:48.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T13:50:48.961 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T13:50:48.961 INFO:tasks.ceph:role: osd.1 2024-09-13T13:50:48.962 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:50:48.962 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T13:50:49.034 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:50:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:50:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:50:49.034 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:50:49.035 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:50:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:50:49.035 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:50:49.035 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:50:49.035 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:50:49.035 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:50:49.489 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-13T13:50:49.491 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-13T13:50:49.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T13:50:49.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T13:50:49.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T13:50:49.679 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-13T13:50:49.679 INFO:tasks.ceph:role: osd.2 2024-09-13T13:50:49.679 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:50:49.679 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T13:50:49.750 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:50:49.750 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:50:49.751 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:50:50.245 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-09-13T13:50:50.247 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-09-13T13:50:50.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T13:50:50.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T13:50:50.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:50.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:50.525+0000 7f761552e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T13:50:50.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:50.529+0000 7f761552e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T13:50:50.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:50.529+0000 7f761552e640 -1 bdev(0x55ba34053800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-13T13:50:50.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:50.529+0000 7f761552e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T13:50:53.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T13:50:53.981 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:54.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:54.070+0000 7fd0d4a1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T13:50:54.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:54.070+0000 7fd0d4a1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T13:50:54.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:54.070+0000 7fd0d4a1c640 -1 bdev(0x55caa2b95800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-13T13:50:54.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:54.071+0000 7fd0d4a1c640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T13:50:57.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T13:50:57.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:50:57.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:57.363+0000 7fef5462c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T13:50:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:57.363+0000 7fef5462c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T13:50:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:57.363+0000 7fef5462c640 -1 bdev(0x562233a21800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-13T13:50:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:50:57.363+0000 7fef5462c640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T13:51:00.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T13:51:00.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T13:51:00.591 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T13:51:00.591 INFO:tasks.ceph:role: osd.3 2024-09-13T13:51:00.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:51:00.591 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:51:00.667 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:51:01.173 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-13T13:51:01.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-13T13:51:01.176 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-13T13:51:01.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T13:51:01.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T13:51:01.416 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T13:51:01.416 INFO:tasks.ceph:role: osd.4 2024-09-13T13:51:01.416 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:51:01.416 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:51:01.489 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:51:01.490 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:51:01.981 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-13T13:51:01.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-13T13:51:01.983 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-13T13:51:02.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T13:51:02.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T13:51:02.196 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-13T13:51:02.197 INFO:tasks.ceph:role: osd.5 2024-09-13T13:51:02.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:51:02.197 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T13:51:02.269 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T13:51:02.797 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-13T13:51:02.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-13T13:51:02.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-13T13:51:02.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T13:51:02.959 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:51:03.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:03.064+0000 7f218ee32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T13:51:03.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:03.069+0000 7f218ee32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T13:51:03.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:03.069+0000 7f218ee32640 -1 bdev(0x55b1f3737800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-13T13:51:03.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:03.069+0000 7f218ee32640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T13:51:06.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T13:51:06.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:06.593+0000 7f6dda0f1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:06.594+0000 7f6dda0f1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:06.594+0000 7f6dda0f1640 -1 bdev(0x563b18191800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-13T13:51:06.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:06.594+0000 7f6dda0f1640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T13:51:10.002 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T13:51:10.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:51:10.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:10.120+0000 7f1969b69640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T13:51:10.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:10.120+0000 7f1969b69640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T13:51:10.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:10.120+0000 7f1969b69640 -1 bdev(0x5638dc485800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-13T13:51:10.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:10.121+0000 7f1969b69640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T13:51:13.530 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T13:51:13.559 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T13:51:13.560 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:51:13.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T13:51:13.593 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:51:13.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T13:51:13.661 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:51:13.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T13:51:13.729 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:13.730 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T13:51:13.760 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:13.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T13:51:13.828 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:13.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T13:51:13.896 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:13.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T13:51:13.963 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:13.963 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T13:51:14.019 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T13:51:14.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T13:51:14.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T13:51:14.063 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-09-13T13:51:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQA6Q+RmhVZiHxAAT7uIGPEodIlgGiCT9n3LYw== 2024-09-13T13:51:14.063 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-09-13T13:51:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQA+Q+Rm5ABBBBAA1syyqOS9GPefY24O2HkVlQ== 2024-09-13T13:51:14.063 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-09-13T13:51:14.063 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBBQ+Rmzw+1FRAA/nJd+vBAItq6vDv96u+emQ== 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQA4Q+RmGhy8AhAABlWSygfOAYdbHSDEzr8sNA== 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBHQ+RmsXrpAxAA9NVmHfrZNy8yfwITOEJ6Tg== 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBKQ+Rmh69+IxAAHwOGLcfAdLEGQmKtVfKSSQ== 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBOQ+RmSu9JBxAAaWFeJbN82/rckHwWr2RgxA== 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-09-13T13:51:14.064 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQA4Q+RmibhWBRAASr5LwmqRmRT68k2moXaMWg== 2024-09-13T13:51:14.084 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-09-13T13:51:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA6Q+RmhVZiHxAAT7uIGPEodIlgGiCT9n3LYw== 2024-09-13T13:51:14.084 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-09-13T13:51:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA+Q+Rm5ABBBBAA1syyqOS9GPefY24O2HkVlQ== 2024-09-13T13:51:14.084 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-09-13T13:51:14.084 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBBQ+Rmzw+1FRAA/nJd+vBAItq6vDv96u+emQ== 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA4Q+RmGhy8AhAABlWSygfOAYdbHSDEzr8sNA== 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBHQ+RmsXrpAxAA9NVmHfrZNy8yfwITOEJ6Tg== 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBKQ+Rmh69+IxAAHwOGLcfAdLEGQmKtVfKSSQ== 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBOQ+RmSu9JBxAAaWFeJbN82/rckHwWr2RgxA== 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-09-13T13:51:14.085 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA4Q+RmibhWBRAASr5LwmqRmRT68k2moXaMWg== 2024-09-13T13:51:14.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.111 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.181 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:51:14.289 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T13:51:14.340 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.342 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.394 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T13:51:14.500 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T13:51:14.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T13:51:14.557 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T13:51:14.557 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T13:51:14.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:14.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T13:51:14.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T13:51:14.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:14.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T13:51:14.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T13:51:14.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:15.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T13:51:15.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-09-13T13:51:15.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:15.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-09-13T13:51:15.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-09-13T13:51:15.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:15.420 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-09-13T13:51:15.444 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-09-13T13:51:15.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:15.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-09-13T13:51:15.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-09-13T13:51:15.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:15.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-09-13T13:51:15.794 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-09-13T13:51:15.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:15.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-09-13T13:51:15.980 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-09-13T13:51:16.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T13:51:16.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-09-13T13:51:16.168 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:51:16.171 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T13:51:16.224 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T13:51:16.224 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T13:51:16.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T13:51:16.228 INFO:tasks.ceph.mon.a:Started 2024-09-13T13:51:16.228 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T13:51:16.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T13:51:16.230 INFO:tasks.ceph.mon.b:Started 2024-09-13T13:51:16.230 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T13:51:16.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T13:51:16.232 INFO:tasks.ceph.mon.c:Started 2024-09-13T13:51:16.232 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-13T13:51:16.233 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-13T13:51:16.234 INFO:tasks.ceph.mon.d:Started 2024-09-13T13:51:16.235 INFO:tasks.ceph.mon.e:Restarting daemon 2024-09-13T13:51:16.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-09-13T13:51:16.237 INFO:tasks.ceph.mon.e:Started 2024-09-13T13:51:16.237 INFO:tasks.ceph.mon.f:Restarting daemon 2024-09-13T13:51:16.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-09-13T13:51:16.266 INFO:tasks.ceph.mon.f:Started 2024-09-13T13:51:16.266 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-13T13:51:16.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-13T13:51:16.268 INFO:tasks.ceph.mon.g:Started 2024-09-13T13:51:16.268 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-13T13:51:16.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-13T13:51:16.270 INFO:tasks.ceph.mon.h:Started 2024-09-13T13:51:16.270 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-13T13:51:16.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-13T13:51:16.272 INFO:tasks.ceph.mon.i:Started 2024-09-13T13:51:16.272 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T13:51:16.273 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T13:51:16.273 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T13:51:16.275 INFO:tasks.ceph.mgr.x:Started 2024-09-13T13:51:16.275 DEBUG:tasks.ceph:set 0 configs 2024-09-13T13:51:16.275 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T13:51:16.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T13:51:16.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.398+0000 7fc2f5156640 10 monclient: get_monmap_and_config 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.398+0000 7fc2f5156640 10 monclient: build_initial_monmap 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.398+0000 7fc2f5156640 10 monclient: monmap: 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:16.398808+0000 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:16.398808+0000 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:16.399 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:16.400 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:16.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.398+0000 7fc2f5156640 1 Processor -- start 2024-09-13T13:51:16.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.398+0000 7fc2f5156640 1 -- start start 2024-09-13T13:51:16.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.398+0000 7fc2f5156640 10 monclient: init 2024-09-13T13:51:16.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient: _add_conns ranks=[6,4,1,8,7,0,3,5,2] 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient(hunting): picked mon.noname-d con 0x7fc2f00a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f009d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient(hunting): picked mon.noname-f con 0x7fc2f00a6c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f009e210 con 0x7fc2f00a6c50 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f00a7ef0 con 0x7fc2f009d150 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f009d970 con 0x7fc2f00a6160 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2f5156640 10 monclient(hunting): authenticate will time out at 756.028224s 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2ee59c640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fc2f009d150 0x7fc2f013e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:54516/0 (socket says 172.21.15.121:54516) 2024-09-13T13:51:16.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.399+0000 7fc2ee59c640 1 -- 172.21.15.121:0/2820835813 learned_addr learned my addr 172.21.15.121:0/2820835813 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:16.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.401+0000 7fc2ee59c640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.121:6791/0 conn(0x7fc2f009d150 legacy=0x7fc2f013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:16.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.401+0000 7fc2eed9d640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.121:6792/0 conn(0x7fc2f00a6160 legacy=0x7fc2f00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:16.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:16.401+0000 7fc2ef59e640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.153:6789/0 conn(0x7fc2f00a6c50 legacy=0x7fc2f0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): tick 2024-09-13T13:51:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): _check_auth_tickets 2024-09-13T13:51:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 monclient(hunting): continuing hunt 2024-09-13T13:51:19.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T13:51:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.121:6792/0 conn(0x7fc2f00a6160 legacy=0x7fc2f00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.121:6791/0 conn(0x7fc2f009d150 legacy=0x7fc2f013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.153:6789/0 conn(0x7fc2f00a6c50 legacy=0x7fc2f0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient: _add_conns ranks=[8,3,5,0,2,7] 2024-09-13T13:51:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): picked mon.noname-e con 0x7fc2d0003960 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:19.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): picked mon.noname-g con 0x7fc2d00075f0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): picked mon.noname-h con 0x7fc2d000b360 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f009e630 con 0x7fc2d00075f0 2024-09-13T13:51:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d0010160 con 0x7fc2d000b360 2024-09-13T13:51:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2d0011400 con 0x7fc2d0003960 2024-09-13T13:51:19.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.399+0000 7fc2ecd99640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.401+0000 7fc2eed9d640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.121:6793/0 conn(0x7fc2d0003960 legacy=0x7fc2d0005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:19.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.402+0000 7fc2ee59c640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.153:6790/0 conn(0x7fc2d00075f0 legacy=0x7fc2d0009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:19.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:19.402+0000 7fc2ef59e640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.153:6791/0 conn(0x7fc2d000b360 legacy=0x7fc2d000d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309557434 0 0) 0x7fc2d0010160 con 0x7fc2d000b360 2024-09-13T13:51:22.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): my global_id is 4109 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2dc003520 con 0x7fc2d000b360 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1020772963 0 0) 0x7fc2f009e630 con 0x7fc2d00075f0 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 10 monclient(hunting): my global_id is 4107 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.366+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d0010160 con 0x7fc2d00075f0 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.367+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788541088 0 0) 0x7fc2d0011400 con 0x7fc2d0003960 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.367+0000 7fc2edd9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.367+0000 7fc2edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:22.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.367+0000 7fc2edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.367+0000 7fc2edd9b640 10 monclient(hunting): my global_id is 4112 2024-09-13T13:51:22.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.367+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2f009e630 con 0x7fc2d0003960 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4081346096 0 0) 0x7fc2d0010160 con 0x7fc2d00075f0 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.121:6793/0 conn(0x7fc2d0003960 legacy=0x7fc2d0005db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.153:6791/0 conn(0x7fc2d000b360 legacy=0x7fc2d000d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: _finish_hunting 0 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 monclient: found mon.noname-g 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2d00126a0 con 0x7fc2d00075f0 2024-09-13T13:51:22.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: _finish_auth 0 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: _check_auth_tickets 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2ef59e640 1 -- 172.21.15.121:0/2820835813 reap_dead start 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2f5156640 5 monclient: authenticate success, global_id 4107 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2f5156640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc2d8001a60 con 0x7fc2d00075f0 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: dump: 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.395 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2f5156640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:22.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2d80048a0 con 0x7fc2d00075f0 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 1 -- 172.21.15.121:0/2820835813 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc2d8004d90 con 0x7fc2d00075f0 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2edd9b640 10 monclient: dump: 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:22.397 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2f5156640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.383+0000 7fc2f5156640 4 set_mon_vals no callback set 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 10 monclient: shutdown 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:22.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 1 -- 172.21.15.121:0/2820835813 >> v1:172.21.15.153:6790/0 conn(0x7fc2d00075f0 legacy=0x7fc2d0009ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 1 -- 172.21.15.121:0/2820835813 shutdown_connections 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 1 -- 172.21.15.121:0/2820835813 >> 172.21.15.121:0/2820835813 conn(0x7fc2f00ab940 msgr2=0x7fc2f00abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 1 -- 172.21.15.121:0/2820835813 shutdown_connections 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.384+0000 7fc2f5156640 1 -- 172.21.15.121:0/2820835813 wait complete. 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.385+0000 7fc2f5156640 10 monclient: build_initial_monmap 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.385+0000 7fc2f5156640 10 monclient: monmap: 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:22.399 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.386+0000 7fc2f5156640 1 Processor -- start 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.386+0000 7fc2f5156640 1 -- start start 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.386+0000 7fc2f5156640 10 monclient: init 2024-09-13T13:51:22.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient: _add_conns ranks=[7,4,8,5,6,1,0,2,3] 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient(hunting): picked mon.noname-i con 0x7fc2d00075f0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient(hunting): picked mon.noname-c con 0x7fc2f0139c60 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient(hunting): picked mon.noname-e con 0x7fc2f01d4190 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f0127490 con 0x7fc2f0139c60 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f0127a60 con 0x7fc2d00075f0 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f0125fa0 con 0x7fc2f01d4190 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2f5156640 10 monclient(hunting): authenticate will time out at 762.016434s 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2eed9d640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7fc2d00075f0 0x7fc2f01244b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.121:37318/0 (socket says 172.21.15.121:37318) 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2ee59c640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fc2f0139c60 0x7fc2f0124bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:54544/0 (socket says 172.21.15.121:54544) 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.387+0000 7fc2eed9d640 1 -- 172.21.15.121:0/63339941 learned_addr learned my addr 172.21.15.121:0/63339941 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:22.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2ee59c640 1 -- 172.21.15.121:0/63339941 >> v1:172.21.15.121:6791/0 conn(0x7fc2f0139c60 legacy=0x7fc2f0124bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2eed9d640 1 -- 172.21.15.121:0/63339941 >> v1:172.21.15.153:6792/0 conn(0x7fc2d00075f0 legacy=0x7fc2f01244b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3271306328 0 0) 0x7fc2f009e630 con 0x7fc2d00075f0 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2e77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2e77fe640 10 monclient(hunting): my global_id is 4111 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.389+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2d0009290 con 0x7fc2d00075f0 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4254062657 0 0) 0x7fc2d0009290 con 0x7fc2d00075f0 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc2f009e630 con 0x7fc2d00075f0 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc2e8002090 con 0x7fc2d00075f0 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:22.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient(hunting): dump: 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:22.403 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1127613327 0 0) 0x7fc2f009e630 con 0x7fc2d00075f0 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 >> v1:172.21.15.121:6793/0 conn(0x7fc2f01d4190 legacy=0x7fc2f01d6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 >> v1:172.21.15.121:6791/0 conn(0x7fc2f0139c60 legacy=0x7fc2f0124bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 monclient: found mon.i 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f01252d0 con 0x7fc2d00075f0 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:22.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 5 monclient: authenticate success, global_id 4111 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 10 monclient: _renew_subs 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f0128300 con 0x7fc2d00075f0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 10 monclient: _renew_subs 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fc2f01d7ca0 con 0x7fc2d00075f0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc2e8003e60 con 0x7fc2d00075f0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc2e8004350 con 0x7fc2d00075f0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 10 monclient: dump: 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:22.405 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:22.406 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2ed59a640 4 set_mon_vals no callback set 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc2e8005b40 con 0x7fc2d00075f0 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.390+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 1972988647 0 0) 0x7fc2e80070f0 con 0x7fc2d00075f0 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.391+0000 7fc2f5156640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.391+0000 7fc2f5156640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.391+0000 7fc2f5156640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:22.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.391+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2d4005080 con 0x7fc2d00075f0 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.393+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fc2f0127a60 con 0x7fc2d00075f0 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.393+0000 7fc2e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:22.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.393+0000 7fc2e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.460+0000 7fc2f5156640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.460+0000 7fc2f5156640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.460+0000 7fc2f5156640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:22.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.460+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 --> v1:172.21.15.153:6792/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc2d4005370 con 0x7fc2d00075f0 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.926+0000 7fc2e77fe640 1 -- 172.21.15.121:0/63339941 <== mon.7 v1:172.21.15.153:6792/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc2e80061d0 con 0x7fc2d00075f0 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.926+0000 7fc2e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.926+0000 7fc2e77fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-13T13:51:22.927 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 10 monclient: shutdown 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 >> v1:172.21.15.153:6792/0 conn(0x7fc2d00075f0 legacy=0x7fc2f01244b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 shutdown_connections 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 >> 172.21.15.121:0/63339941 conn(0x7fc2f00ab940 msgr2=0x7fc2f01c06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 shutdown_connections 2024-09-13T13:51:22.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:22.928+0000 7fc2f5156640 1 -- 172.21.15.121:0/63339941 wait complete. 2024-09-13T13:51:22.942 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T13:51:22.942 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T13:51:22.943 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:51:22.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T13:51:22.973 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:51:22.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T13:51:23.040 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-09-13T13:51:23.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T13:51:23.105 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:23.106 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T13:51:23.131 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:23.131 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T13:51:23.196 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-13T13:51:23.197 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T13:51:23.262 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new b6c3c9e1-ad77-4e07-a73a-95b4956715e7 0 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.402+0000 7f93db295640 10 monclient: get_monmap_and_config 2024-09-13T13:51:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.402+0000 7f93db295640 10 monclient: build_initial_monmap 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.402+0000 7f93db295640 10 monclient: monmap: 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:51:23.403593+0000 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:51:23.403593+0000 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:23.405 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.403+0000 7f93db295640 1 Processor -- start 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.403+0000 7f93db295640 1 -- start start 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.403+0000 7f93db295640 10 monclient: init 2024-09-13T13:51:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient: _add_conns ranks=[2,0,4,1,8,7,6,5,3] 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient(hunting): picked mon.noname-b con 0x7f93d4139050 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient(hunting): picked mon.noname-a con 0x7f93d4139b40 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient(hunting): picked mon.noname-c con 0x7f93d40a4eb0 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d406ebb0 con 0x7f93d4139b40 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d40a5b10 con 0x7f93d4139050 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d406e8d0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93db295640 10 monclient(hunting): authenticate will time out at 720.927185s 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93d9031640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f93d4139050 0x7f93d4139430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.153:37280/0 (socket says 172.21.15.153:37280) 2024-09-13T13:51:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.404+0000 7f93d9031640 1 -- 172.21.15.153:0/893336334 learned_addr learned my addr 172.21.15.153:0/893336334 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93d9031640 1 -- 172.21.15.153:0/893336334 >> v1:172.21.15.121:6790/0 conn(0x7f93d4139050 legacy=0x7f93d4139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93d9832640 1 -- 172.21.15.153:0/893336334 >> v1:172.21.15.121:6791/0 conn(0x7f93d40a4eb0 legacy=0x7f93d41419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93d8830640 1 -- 172.21.15.153:0/893336334 >> v1:172.21.15.121:6789/0 conn(0x7f93d4139b40 legacy=0x7f93d413e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848296111 0 0) 0x7f93d40a5b10 con 0x7f93d4139050 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): my global_id is 4106 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93c0003520 con 0x7f93d4139050 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2102273946 0 0) 0x7f93d406e8d0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): my global_id is 4117 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93d40a5b10 con 0x7f93d40a4eb0 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402836333 0 0) 0x7f93d406ebb0 con 0x7f93d4139b40 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 10 monclient(hunting): my global_id is 4104 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.406+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93d406e8d0 con 0x7f93d4139b40 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1240222734 0 0) 0x7f93c0003520 con 0x7f93d4139050 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 >> v1:172.21.15.121:6791/0 conn(0x7f93d40a4eb0 legacy=0x7f93d41419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 >> v1:172.21.15.121:6789/0 conn(0x7f93d4139b40 legacy=0x7f93d413e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 monclient: found mon.noname-b 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d406dc90 con 0x7f93d4139050 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: _finish_auth 0 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f93c4001ae0 con 0x7f93d4139050 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: dump: 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:23.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 5 monclient: authenticate success, global_id 4106 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93c40048a0 con 0x7f93d4139050 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 1 -- 172.21.15.153:0/893336334 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f93c4004e10 con 0x7f93d4139050 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 4 set_mon_vals no callback set 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93cbfff640 10 monclient: dump: 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 10 monclient: shutdown 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 1 -- 172.21.15.153:0/893336334 >> v1:172.21.15.121:6790/0 conn(0x7f93d4139050 legacy=0x7f93d4139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 1 -- 172.21.15.153:0/893336334 shutdown_connections 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.407+0000 7f93db295640 1 -- 172.21.15.153:0/893336334 >> 172.21.15.153:0/893336334 conn(0x7f93d40a9680 msgr2=0x7f93d40a9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 1 -- 172.21.15.153:0/893336334 shutdown_connections 2024-09-13T13:51:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 1 -- 172.21.15.153:0/893336334 wait complete. 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 10 monclient: build_initial_monmap 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 10 monclient: monmap: 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 1 Processor -- start 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 1 -- start start 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.408+0000 7f93db295640 10 monclient: init 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient: _add_conns ranks=[7,0,6,1,5,2,8,4,3] 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient(hunting): picked mon.noname-i con 0x7f93d40a4eb0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient(hunting): picked mon.noname-a con 0x7f93d4139050 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient(hunting): picked mon.noname-d con 0x7f93d4139b40 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d40a4af0 con 0x7f93d4139050 2024-09-13T13:51:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d40a3550 con 0x7f93d4139b40 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d40a8cd0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93db295640 10 monclient(hunting): authenticate will time out at 720.932452s 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93d9031640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7f93d40a4eb0 0x7f93d41384b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.153:57722/0 (socket says 172.21.15.153:57722) 2024-09-13T13:51:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.409+0000 7f93d9031640 1 -- 172.21.15.153:0/2762423273 learned_addr learned my addr 172.21.15.153:0/2762423273 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93d9031640 1 -- 172.21.15.153:0/2762423273 >> v1:172.21.15.153:6792/0 conn(0x7f93d40a4eb0 legacy=0x7f93d41384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93d9832640 1 -- 172.21.15.153:0/2762423273 >> v1:172.21.15.121:6792/0 conn(0x7f93d4139b40 legacy=0x7f93d41d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93d8830640 1 -- 172.21.15.153:0/2762423273 >> v1:172.21.15.121:6789/0 conn(0x7f93d4139050 legacy=0x7f93d41d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2960334064 0 0) 0x7f93d40a8cd0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93c9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93c9ffb640 10 monclient(hunting): my global_id is 4120 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.411+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93b8003520 con 0x7f93d40a4eb0 2024-09-13T13:51:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105938695 0 0) 0x7f93d40a3550 con 0x7f93d4139b40 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): my global_id is 4119 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93d40a8cd0 con 0x7f93d4139b40 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3296844786 0 0) 0x7f93d40a4af0 con 0x7f93d4139050 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): my global_id is 4113 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93d40a3550 con 0x7f93d4139050 2024-09-13T13:51:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2739136757 0 0) 0x7f93b8003520 con 0x7f93d40a4eb0 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f93d40a4af0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f93c4004d90 con 0x7f93d40a4eb0 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): dump: 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3734296580 0 0) 0x7f93d40a4af0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 >> v1:172.21.15.121:6792/0 conn(0x7f93d4139b40 legacy=0x7f93d41d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 >> v1:172.21.15.121:6789/0 conn(0x7f93d4139050 legacy=0x7f93d41d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 monclient: found mon.i 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d4124670 con 0x7f93d40a4eb0 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 5 monclient: authenticate success, global_id 4120 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 10 monclient: _renew_subs 2024-09-13T13:51:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f93d4138bc0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93c4001d30 con 0x7f93d40a4eb0 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f93c4001a80 con 0x7f93d40a4eb0 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 10 monclient: dump: 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f93c4005ad0 con 0x7f93d40a4eb0 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93cb7fe640 4 set_mon_vals no callback set 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 10 monclient: _renew_subs 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f93d41cda70 con 0x7f93d40a4eb0 2024-09-13T13:51:23.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.412+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 339553649 0 0) 0x7f93c4006380 con 0x7f93d40a4eb0 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.413+0000 7f93db295640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.413+0000 7f93db295640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.413+0000 7f93db295640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.413+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d4074370 con 0x7f93d40a4eb0 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.415+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f93d4074370 con 0x7f93d40a4eb0 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.415+0000 7f93c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:23.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.415+0000 7f93c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:23.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.486+0000 7f93db295640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b6c3c9e1-ad77-4e07-a73a-95b4956715e7", "id": 0}] 2024-09-13T13:51:23.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.486+0000 7f93db295640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b6c3c9e1-ad77-4e07-a73a-95b4956715e7", "id": 0}] 2024-09-13T13:51:23.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.486+0000 7f93db295640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:51:23.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:23.486+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 --> v1:172.21.15.153:6792/0 -- mon_command({"prefix": "osd new", "uuid": "b6c3c9e1-ad77-4e07-a73a-95b4956715e7", "id": 0} v 0) -- 0x7f93d40a9190 con 0x7f93d40a4eb0 2024-09-13T13:51:24.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.133+0000 7f93c9ffb640 1 -- 172.21.15.153:0/2762423273 <== mon.7 v1:172.21.15.153:6792/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6c3c9e1-ad77-4e07-a73a-95b4956715e7", "id": 0}]=0 v3) ==== 112+0+2 (unknown 122214303 0 732707515) 0x7f93c4006580 con 0x7f93d40a4eb0 2024-09-13T13:51:24.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.133+0000 7f93c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b6c3c9e1-ad77-4e07-a73a-95b4956715e7", "id": 0}] 2024-09-13T13:51:24.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.133+0000 7f93c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:24.135 INFO:teuthology.orchestra.run.smithi153.stdout:0 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 10 monclient: shutdown 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 >> v1:172.21.15.153:6792/0 conn(0x7f93d40a4eb0 legacy=0x7f93d41384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 shutdown_connections 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 >> 172.21.15.153:0/2762423273 conn(0x7f93d40a9680 msgr2=0x7f93d40a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 shutdown_connections 2024-09-13T13:51:24.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.135+0000 7f93db295640 1 -- 172.21.15.153:0/2762423273 wait complete. 2024-09-13T13:51:24.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 6d5efbfa-eca0-4806-9229-69bbc58aa605 1 2024-09-13T13:51:24.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.242+0000 7fc7537bb640 10 monclient: get_monmap_and_config 2024-09-13T13:51:24.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.242+0000 7fc7537bb640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.242+0000 7fc7537bb640 10 monclient: monmap: 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:51:24.243417+0000 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:51:24.243417+0000 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.244 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.245 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 1 Processor -- start 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 1 -- start start 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient: init 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient: _add_conns ranks=[4,7,3,1,6,5,2,0,8] 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient(hunting): picked mon.noname-c con 0x7fc74c0a6160 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:24.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient(hunting): picked mon.noname-i con 0x7fc74c09d150 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient(hunting): picked mon.noname-g con 0x7fc74c0a6c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c09e210 con 0x7fc74c0a6c50 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c0a7ef0 con 0x7fc74c0a6160 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c09d970 con 0x7fc74c09d150 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.243+0000 7fc7537bb640 10 monclient(hunting): authenticate will time out at 721.766966s 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.244+0000 7fc751d58640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fc74c0a6c50 0x7fc74c141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:44118/0 (socket says 172.21.15.153:44118) 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.244+0000 7fc751d58640 1 -- 172.21.15.153:0/3105726277 learned_addr learned my addr 172.21.15.153:0/3105726277 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.244+0000 7fc750d56640 1 --1- 172.21.15.153:0/3105726277 >> v1:172.21.15.153:6792/0 conn(0x7fc74c09d150 0x7fc74c13e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.153:57732/0 (socket says 172.21.15.153:57732) 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc750d56640 1 -- 172.21.15.153:0/3105726277 >> v1:172.21.15.153:6792/0 conn(0x7fc74c09d150 legacy=0x7fc74c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc751d58640 1 -- 172.21.15.153:0/3105726277 >> v1:172.21.15.153:6790/0 conn(0x7fc74c0a6c50 legacy=0x7fc74c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2402723442 0 0) 0x7fc74c09e210 con 0x7fc74c0a6c50 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): my global_id is 4125 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc738003520 con 0x7fc74c0a6c50 2024-09-13T13:51:24.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889289627 0 0) 0x7fc74c09d970 con 0x7fc74c09d150 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient(hunting): my global_id is 4129 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc74c09e210 con 0x7fc74c09d150 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc751557640 1 -- 172.21.15.153:0/3105726277 >> v1:172.21.15.121:6791/0 conn(0x7fc74c0a6160 legacy=0x7fc74c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 732385378 0 0) 0x7fc738003520 con 0x7fc74c0a6c50 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 >> v1:172.21.15.153:6792/0 conn(0x7fc74c09d150 legacy=0x7fc74c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 >> v1:172.21.15.121:6791/0 conn(0x7fc74c0a6160 legacy=0x7fc74c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 monclient: found mon.noname-g 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c09e630 con 0x7fc74c0a6c50 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc744001d90 con 0x7fc74c0a6c50 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc7537bb640 5 monclient: authenticate success, global_id 4125 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: dump: 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.253 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc7537bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc744002060 con 0x7fc74c0a6c50 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 1 -- 172.21.15.153:0/3105726277 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc744004f60 con 0x7fc74c0a6c50 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc7537bb640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc7537bb640 4 set_mon_vals no callback set 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.246+0000 7fc743fff640 10 monclient: dump: 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 10 monclient: shutdown 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 1 -- 172.21.15.153:0/3105726277 >> v1:172.21.15.153:6790/0 conn(0x7fc74c0a6c50 legacy=0x7fc74c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 1 -- 172.21.15.153:0/3105726277 shutdown_connections 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 1 -- 172.21.15.153:0/3105726277 >> 172.21.15.153:0/3105726277 conn(0x7fc74c0ab940 msgr2=0x7fc74c0abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 1 -- 172.21.15.153:0/3105726277 shutdown_connections 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.247+0000 7fc7537bb640 1 -- 172.21.15.153:0/3105726277 wait complete. 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient: monmap: 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.257 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 1 Processor -- start 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 1 -- start start 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient: init 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient: _add_conns ranks=[1,6,3,4,2,8,7,0,5] 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient(hunting): picked mon.noname-f con 0x7fc74c09d150 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient(hunting): picked mon.noname-d con 0x7fc74c0a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient(hunting): picked mon.noname-g con 0x7fc74c0a6c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c1248d0 con 0x7fc74c09d150 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c0a4650 con 0x7fc74c0a6c50 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc74c126950 con 0x7fc74c0a6160 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc7537bb640 10 monclient(hunting): authenticate will time out at 721.771901s 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc751557640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc74c09d150 0x7fc74c139270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45520/0 (socket says 172.21.15.153:45520) 2024-09-13T13:51:24.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.248+0000 7fc751557640 1 -- 172.21.15.153:0/246750495 learned_addr learned my addr 172.21.15.153:0/246750495 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc751557640 1 -- 172.21.15.153:0/246750495 >> v1:172.21.15.153:6789/0 conn(0x7fc74c09d150 legacy=0x7fc74c139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc751d58640 1 -- 172.21.15.153:0/246750495 >> v1:172.21.15.153:6790/0 conn(0x7fc74c0a6c50 legacy=0x7fc74c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3912928408 0 0) 0x7fc74c1248d0 con 0x7fc74c09d150 2024-09-13T13:51:24.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): my global_id is 4123 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc730003520 con 0x7fc74c09d150 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4176297861 0 0) 0x7fc74c09e630 con 0x7fc74c0a6c50 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): my global_id is 4134 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc74c1248d0 con 0x7fc74c0a6c50 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc750d56640 1 -- 172.21.15.153:0/246750495 >> v1:172.21.15.121:6792/0 conn(0x7fc74c0a6160 legacy=0x7fc74c1d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1483381898 0 0) 0x7fc730003520 con 0x7fc74c09d150 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc74c09e630 con 0x7fc74c09d150 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc748003610 con 0x7fc74c09d150 2024-09-13T13:51:24.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): dump: 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2564743330 0 0) 0x7fc74c1248d0 con 0x7fc74c0a6c50 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc730003520 con 0x7fc74c0a6c50 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc744001d70 con 0x7fc74c0a6c50 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): dump: 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1971978116 0 0) 0x7fc74c126950 con 0x7fc74c0a6160 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient(hunting): my global_id is 4128 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc74c1248d0 con 0x7fc74c0a6160 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3172368442 0 0) 0x7fc74c09e630 con 0x7fc74c09d150 2024-09-13T13:51:24.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 >> v1:172.21.15.121:6792/0 conn(0x7fc74c0a6160 legacy=0x7fc74c1d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 >> v1:172.21.15.153:6790/0 conn(0x7fc74c0a6c50 legacy=0x7fc74c1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 monclient: found mon.f 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74c126c30 con 0x7fc74c09d150 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 5 monclient: authenticate success, global_id 4123 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 10 monclient: _renew_subs 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc74c0a8f10 con 0x7fc74c09d150 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc748003bc0 con 0x7fc74c09d150 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc7480040f0 con 0x7fc74c09d150 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 10 monclient: dump: 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 10 monclient: _renew_subs 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7437fe640 4 set_mon_vals no callback set 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc748004310 con 0x7fc74c09d150 2024-09-13T13:51:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.251+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc74c125430 con 0x7fc74c09d150 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.252+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 2957787145 0 0) 0x7fc748006370 con 0x7fc74c09d150 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.252+0000 7fc7537bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.253+0000 7fc7537bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.253+0000 7fc7537bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.253+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74c1265d0 con 0x7fc74c09d150 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.254+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fc748005f00 con 0x7fc74c09d150 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.255+0000 7fc741ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.255+0000 7fc741ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:24.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.325+0000 7fc7537bb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6d5efbfa-eca0-4806-9229-69bbc58aa605", "id": 1}] 2024-09-13T13:51:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.325+0000 7fc7537bb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6d5efbfa-eca0-4806-9229-69bbc58aa605", "id": 1}] 2024-09-13T13:51:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.325+0000 7fc7537bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.325+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6d5efbfa-eca0-4806-9229-69bbc58aa605", "id": 1} v 0) -- 0x7fc74c074b10 con 0x7fc74c09d150 2024-09-13T13:51:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.329+0000 7fc741ffb640 1 -- 172.21.15.153:0/246750495 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6d5efbfa-eca0-4806-9229-69bbc58aa605", "id": 1}]=0 v4) ==== 112+0+2 (unknown 1777158144 0 940483276) 0x7fc7480069f0 con 0x7fc74c09d150 2024-09-13T13:51:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.329+0000 7fc741ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6d5efbfa-eca0-4806-9229-69bbc58aa605", "id": 1}] 2024-09-13T13:51:24.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.329+0000 7fc741ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:24.331 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 10 monclient: shutdown 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 >> v1:172.21.15.153:6789/0 conn(0x7fc74c09d150 legacy=0x7fc74c139270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 shutdown_connections 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 >> 172.21.15.153:0/246750495 conn(0x7fc74c0ab940 msgr2=0x7fc74c09f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 shutdown_connections 2024-09-13T13:51:24.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.330+0000 7fc7537bb640 1 -- 172.21.15.153:0/246750495 wait complete. 2024-09-13T13:51:24.341 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new fba721b0-d5d8-4735-b8e2-a7cf81067ae1 2 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.440+0000 7fbbb3bfb640 10 monclient: get_monmap_and_config 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.440+0000 7fbbb3bfb640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.440+0000 7fbbb3bfb640 10 monclient: monmap: 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:51:24.441598+0000 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:51:24.441598+0000 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.442 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.441+0000 7fbbb3bfb640 1 Processor -- start 2024-09-13T13:51:24.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.441+0000 7fbbb3bfb640 1 -- start start 2024-09-13T13:51:24.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.441+0000 7fbbb3bfb640 10 monclient: init 2024-09-13T13:51:24.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient: _add_conns ranks=[6,1,7,5,0,8,3,4,2] 2024-09-13T13:51:24.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient(hunting): picked mon.noname-d con 0x7fbbac0a3e30 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient(hunting): picked mon.noname-f con 0x7fbbac0a4210 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient(hunting): picked mon.noname-i con 0x7fbbac13d050 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac0737f0 con 0x7fbbac0a4210 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac0734a0 con 0x7fbbac0a3e30 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac0a5b60 con 0x7fbbac13d050 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb3bfb640 10 monclient(hunting): authenticate will time out at 721.965224s 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb2198640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7fbbac13d050 0x7fbbac13f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.153:57734/0 (socket says 172.21.15.153:57734) 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.442+0000 7fbbb2198640 1 -- 172.21.15.153:0/3557303841 learned_addr learned my addr 172.21.15.153:0/3557303841 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb2198640 1 -- 172.21.15.153:0/3557303841 >> v1:172.21.15.153:6792/0 conn(0x7fbbac13d050 legacy=0x7fbbac13f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb1196640 1 -- 172.21.15.153:0/3557303841 >> v1:172.21.15.153:6789/0 conn(0x7fbbac0a4210 legacy=0x7fbbac0a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746774919 0 0) 0x7fbbac0a5b60 con 0x7fbbac13d050 2024-09-13T13:51:24.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): my global_id is 4138 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb94003520 con 0x7fbbac13d050 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 402733866 0 0) 0x7fbbac0737f0 con 0x7fbbac0a4210 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient(hunting): my global_id is 4132 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbac0a5b60 con 0x7fbbac0a4210 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb1997640 1 -- 172.21.15.153:0/3557303841 >> v1:172.21.15.121:6792/0 conn(0x7fbbac0a3e30 legacy=0x7fbbac073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 34605397 0 0) 0x7fbb94003520 con 0x7fbbac13d050 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 >> v1:172.21.15.121:6792/0 conn(0x7fbbac0a3e30 legacy=0x7fbbac073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 >> v1:172.21.15.153:6789/0 conn(0x7fbbac0a4210 legacy=0x7fbbac0a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 monclient: found mon.noname-i 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.153:6792/0 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 --> v1:172.21.15.153:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac143b90 con 0x7fbbac13d050 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb3bfb640 5 monclient: authenticate success, global_id 4138 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbb9c001ae0 con 0x7fbbac13d050 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb3bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.444+0000 7fbbb0995640 10 monclient: dump: 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.451 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 <== mon.7 v1:172.21.15.153:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb9c0048a0 con 0x7fbbac13d050 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb0995640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb0995640 1 -- 172.21.15.153:0/3557303841 <== mon.7 v1:172.21.15.153:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbb9c004e10 con 0x7fbbac13d050 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 4 set_mon_vals no callback set 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb0995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb0995640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb0995640 10 monclient: dump: 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.453 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.454 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 10 monclient: shutdown 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/3557303841 >> v1:172.21.15.153:6792/0 conn(0x7fbbac13d050 legacy=0x7fbbac13f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/3557303841 shutdown_connections 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/3557303841 >> 172.21.15.153:0/3557303841 conn(0x7fbbac0a9430 msgr2=0x7fbbac0a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/3557303841 shutdown_connections 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.445+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/3557303841 wait complete. 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.446+0000 7fbbb3bfb640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.446+0000 7fbbb3bfb640 10 monclient: monmap: 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-13T13:51:24.455 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.446+0000 7fbbb3bfb640 1 Processor -- start 2024-09-13T13:51:24.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.446+0000 7fbbb3bfb640 1 -- start start 2024-09-13T13:51:24.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.446+0000 7fbbb3bfb640 10 monclient: init 2024-09-13T13:51:24.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient: _add_conns ranks=[1,6,8,2,5,7,4,3,0] 2024-09-13T13:51:24.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient(hunting): picked mon.noname-f con 0x7fbbac0a3e30 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:24.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient(hunting): picked mon.noname-d con 0x7fbbac0a4210 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:24.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient(hunting): picked mon.noname-e con 0x7fbbac13d050 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac09f060 con 0x7fbbac0a3e30 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac137a10 con 0x7fbbac0a4210 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbac13c6c0 con 0x7fbbac13d050 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb3bfb640 10 monclient(hunting): authenticate will time out at 721.970464s 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb1997640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbbac0a3e30 0x7fbbac136bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45550/0 (socket says 172.21.15.153:45550) 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.447+0000 7fbbb1997640 1 -- 172.21.15.153:0/2458501256 learned_addr learned my addr 172.21.15.153:0/2458501256 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbb1997640 1 -- 172.21.15.153:0/2458501256 >> v1:172.21.15.153:6789/0 conn(0x7fbbac0a3e30 legacy=0x7fbbac136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbb1196640 1 -- 172.21.15.153:0/2458501256 >> v1:172.21.15.121:6792/0 conn(0x7fbbac0a4210 legacy=0x7fbbac137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 627953547 0 0) 0x7fbbac09f060 con 0x7fbbac0a3e30 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): my global_id is 4141 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb98003520 con 0x7fbbac0a3e30 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbb2198640 1 -- 172.21.15.153:0/2458501256 >> v1:172.21.15.121:6793/0 conn(0x7fbbac13d050 legacy=0x7fbbac1d5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942808156 0 0) 0x7fbbac137a10 con 0x7fbbac0a4210 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.449+0000 7fbbaa7fc640 10 monclient(hunting): my global_id is 4146 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbac09f060 con 0x7fbbac0a4210 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2427975596 0 0) 0x7fbbac13c6c0 con 0x7fbbac13d050 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): my global_id is 4130 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbac137a10 con 0x7fbbac13d050 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1232115269 0 0) 0x7fbb98003520 con 0x7fbbac0a3e30 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbbac13c6c0 con 0x7fbbac0a3e30 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbba4003610 con 0x7fbbac0a3e30 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.464 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 253084252 0 0) 0x7fbbac09f060 con 0x7fbbac0a4210 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb98003520 con 0x7fbbac0a4210 2024-09-13T13:51:24.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbba00016f0 con 0x7fbbac0a4210 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.466 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4131899304 0 0) 0x7fbbac13c6c0 con 0x7fbbac0a3e30 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 >> v1:172.21.15.121:6793/0 conn(0x7fbbac13d050 legacy=0x7fbbac1d5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 >> v1:172.21.15.121:6792/0 conn(0x7fbbac0a4210 legacy=0x7fbbac137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 monclient: found mon.f 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac13ff30 con 0x7fbbac0a3e30 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 5 monclient: authenticate success, global_id 4141 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 10 monclient: _renew_subs 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbbac09ed80 con 0x7fbbac0a3e30 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 10 monclient: _renew_subs 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbbac09f850 con 0x7fbbac0a3e30 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbba4003b90 con 0x7fbbac0a3e30 2024-09-13T13:51:24.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbabfff640 4 set_mon_vals no callback set 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbba40040c0 con 0x7fbbac0a3e30 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 10 monclient: dump: 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.469 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbba4005b50 con 0x7fbbac0a3e30 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.450+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 4230350749 0 0) 0x7fbba4006e20 con 0x7fbbac0a3e30 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.451+0000 7fbbb3bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.451+0000 7fbbb3bfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.451+0000 7fbbb3bfb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.451+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb8c005080 con 0x7fbbac0a3e30 2024-09-13T13:51:24.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.453+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fbba4004400 con 0x7fbbac0a3e30 2024-09-13T13:51:24.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.453+0000 7fbbaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.453+0000 7fbbaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.523+0000 7fbbb3bfb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fba721b0-d5d8-4735-b8e2-a7cf81067ae1", "id": 2}] 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.523+0000 7fbbb3bfb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fba721b0-d5d8-4735-b8e2-a7cf81067ae1", "id": 2}] 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.523+0000 7fbbb3bfb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:24.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.523+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fba721b0-d5d8-4735-b8e2-a7cf81067ae1", "id": 2} v 0) -- 0x7fbb8c005370 con 0x7fbbac0a3e30 2024-09-13T13:51:24.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.529+0000 7fbbaa7fc640 1 -- 172.21.15.153:0/2458501256 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fba721b0-d5d8-4735-b8e2-a7cf81067ae1", "id": 2}]=0 v5) ==== 112+0+2 (unknown 920961773 0 216599125) 0x7fbba4006410 con 0x7fbbac0a3e30 2024-09-13T13:51:24.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.529+0000 7fbbaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fba721b0-d5d8-4735-b8e2-a7cf81067ae1", "id": 2}] 2024-09-13T13:51:24.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.529+0000 7fbbaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:24.530 INFO:teuthology.orchestra.run.smithi153.stdout:2 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 10 monclient: shutdown 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 >> v1:172.21.15.153:6789/0 conn(0x7fbbac0a3e30 legacy=0x7fbbac136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 shutdown_connections 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 >> 172.21.15.153:0/2458501256 conn(0x7fbbac0a9430 msgr2=0x7fbbac0a7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 shutdown_connections 2024-09-13T13:51:24.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.530+0000 7fbbb3bfb640 1 -- 172.21.15.153:0/2458501256 wait complete. 2024-09-13T13:51:24.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 9bbad653-ff80-4a41-a571-666779269a6f 3 2024-09-13T13:51:24.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.639+0000 7f5e1e8b6640 10 monclient: get_monmap_and_config 2024-09-13T13:51:24.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.639+0000 7f5e1e8b6640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.639+0000 7f5e1e8b6640 10 monclient: monmap: 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:51:24.640418+0000 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:51:24.640418+0000 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.643 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 1 Processor -- start 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 1 -- start start 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 10 monclient: init 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 10 monclient: _add_conns ranks=[1,8,5,0,6,2,3,7,4] 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 10 monclient(hunting): picked mon.noname-f con 0x7f5e180a6160 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 10 monclient(hunting): picked mon.noname-e con 0x7f5e1809d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 10 monclient(hunting): picked mon.noname-h con 0x7f5e180a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.640+0000 7f5e1e8b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1809e210 con 0x7f5e180a6160 2024-09-13T13:51:24.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e1e8b6640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e180a7ef0 con 0x7f5e180a6c50 2024-09-13T13:51:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e1e8b6640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1809d970 con 0x7f5e1809d150 2024-09-13T13:51:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e1e8b6640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e1e8b6640 10 monclient(hunting): authenticate will time out at 722.164026s 2024-09-13T13:51:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e17fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5e180a6160 0x7f5e180a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45558/0 (socket says 172.21.15.153:45558) 2024-09-13T13:51:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e17fff640 1 -- 172.21.15.153:0/859451839 learned_addr learned my addr 172.21.15.153:0/859451839 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.641+0000 7f5e1ce53640 1 --1- 172.21.15.153:0/859451839 >> v1:172.21.15.153:6791/0 conn(0x7f5e180a6c50 0x7f5e18141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.153:42908/0 (socket says 172.21.15.153:42908) 2024-09-13T13:51:24.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e1ce53640 1 -- 172.21.15.153:0/859451839 >> v1:172.21.15.153:6791/0 conn(0x7f5e180a6c50 legacy=0x7f5e18141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e17fff640 1 -- 172.21.15.153:0/859451839 >> v1:172.21.15.153:6789/0 conn(0x7f5e180a6160 legacy=0x7f5e180a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3933857810 0 0) 0x7f5e180a7ef0 con 0x7f5e180a6c50 2024-09-13T13:51:24.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): my global_id is 4118 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e04003520 con 0x7f5e180a6c50 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e177fe640 1 -- 172.21.15.153:0/859451839 >> v1:172.21.15.121:6793/0 conn(0x7f5e1809d150 legacy=0x7f5e1813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831692760 0 0) 0x7f5e1809e210 con 0x7f5e180a6160 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient(hunting): my global_id is 4150 2024-09-13T13:51:24.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e180a7ef0 con 0x7f5e180a6160 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3176120886 0 0) 0x7f5e04003520 con 0x7f5e180a6c50 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 >> v1:172.21.15.121:6793/0 conn(0x7f5e1809d150 legacy=0x7f5e1813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 >> v1:172.21.15.153:6789/0 conn(0x7f5e180a6160 legacy=0x7f5e180a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 monclient: found mon.noname-h 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.153:6791/0 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1809e630 con 0x7f5e180a6c50 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5e10001e60 con 0x7f5e180a6c50 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e1e8b6640 5 monclient: authenticate success, global_id 4118 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e1e8b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: dump: 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.659 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e1e8b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 <== mon.5 v1:172.21.15.153:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e10004b60 con 0x7f5e180a6c50 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e1e8b6640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 1 -- 172.21.15.153:0/859451839 <== mon.5 v1:172.21.15.153:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5e100050d0 con 0x7f5e180a6c50 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e1e8b6640 4 set_mon_vals no callback set 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.643+0000 7f5e16ffd640 10 monclient: dump: 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.661 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 10 monclient: shutdown 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/859451839 >> v1:172.21.15.153:6791/0 conn(0x7f5e180a6c50 legacy=0x7f5e18141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/859451839 shutdown_connections 2024-09-13T13:51:24.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/859451839 >> 172.21.15.153:0/859451839 conn(0x7f5e180ab940 msgr2=0x7f5e180abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/859451839 shutdown_connections 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/859451839 wait complete. 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.644+0000 7f5e1e8b6640 10 monclient: monmap: 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.663 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 1 Processor -- start 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 1 -- start start 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient: init 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient: _add_conns ranks=[0,8,5,1,2,6,3,4,7] 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient(hunting): picked mon.noname-a con 0x7f5e1809d150 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient(hunting): picked mon.noname-e con 0x7f5e180a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient(hunting): picked mon.noname-h con 0x7f5e180a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e181248e0 con 0x7f5e1809d150 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e180a8a10 con 0x7f5e180a6c50 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e180a45a0 con 0x7f5e180a6160 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.645+0000 7f5e1e8b6640 10 monclient(hunting): authenticate will time out at 722.168970s 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.646+0000 7f5e1ce53640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f5e180a6c50 0x7f5e181d6140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.153:42910/0 (socket says 172.21.15.153:42910) 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.646+0000 7f5e1ce53640 1 -- 172.21.15.153:0/2973468117 learned_addr learned my addr 172.21.15.153:0/2973468117 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e1ce53640 1 -- 172.21.15.153:0/2973468117 >> v1:172.21.15.153:6791/0 conn(0x7f5e180a6c50 legacy=0x7f5e181d6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2511579604 0 0) 0x7f5e180a8a10 con 0x7f5e180a6c50 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 10 monclient(hunting): my global_id is 4127 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e177fe640 1 -- 172.21.15.153:0/2973468117 >> v1:172.21.15.121:6793/0 conn(0x7f5e180a6160 legacy=0x7f5e181d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5df8003520 con 0x7f5e180a6c50 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 DispatchQueue::entry inject delay of 0.002000 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.648+0000 7f5e17fff640 1 -- 172.21.15.153:0/2973468117 >> v1:172.21.15.121:6789/0 conn(0x7f5e1809d150 legacy=0x7f5e18139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204852202 0 0) 0x7f5e180a45a0 con 0x7f5e180a6160 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): my global_id is 4148 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e180a8a10 con 0x7f5e180a6160 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 726931458 0 0) 0x7f5df8003520 con 0x7f5e180a6c50 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5e180a45a0 con 0x7f5e180a6c50 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1499836072 0 0) 0x7f5e181248e0 con 0x7f5e1809d150 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): my global_id is 4122 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5df8003520 con 0x7f5e1809d150 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5e100020b0 con 0x7f5e180a6c50 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): dump: 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.667 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.668 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2942271046 0 0) 0x7f5e180a45a0 con 0x7f5e180a6c50 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 >> v1:172.21.15.121:6793/0 conn(0x7f5e180a6160 legacy=0x7f5e181d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 >> v1:172.21.15.121:6789/0 conn(0x7f5e1809d150 legacy=0x7f5e18139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 monclient: found mon.h 2024-09-13T13:51:24.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e18126c40 con 0x7f5e180a6c50 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 5 monclient: authenticate success, global_id 4127 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 10 monclient: _renew_subs 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5e181263a0 con 0x7f5e180a6c50 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 10 monclient: _renew_subs 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f5e18125620 con 0x7f5e180a6c50 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5e10005300 con 0x7f5e180a6c50 2024-09-13T13:51:24.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5e10005870 con 0x7f5e180a6c50 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e167fc640 4 set_mon_vals no callback set 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 10 monclient: dump: 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.671 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5e10005e20 con 0x7f5e180a6c50 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.650+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 1424710499 0 0) 0x7f5e10007540 con 0x7f5e180a6c50 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.651+0000 7f5e1e8b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.651+0000 7f5e1e8b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.651+0000 7f5e1e8b6640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:24.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.651+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dfc005080 con 0x7f5e180a6c50 2024-09-13T13:51:24.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.653+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f5e100070b0 con 0x7f5e180a6c50 2024-09-13T13:51:24.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.653+0000 7f5e14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.653+0000 7f5e14ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.724+0000 7f5e1e8b6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9bbad653-ff80-4a41-a571-666779269a6f", "id": 3}] 2024-09-13T13:51:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.724+0000 7f5e1e8b6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9bbad653-ff80-4a41-a571-666779269a6f", "id": 3}] 2024-09-13T13:51:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.724+0000 7f5e1e8b6640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.724+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "osd new", "uuid": "9bbad653-ff80-4a41-a571-666779269a6f", "id": 3} v 0) -- 0x7f5dfc005370 con 0x7f5e180a6c50 2024-09-13T13:51:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.729+0000 7f5e14ff9640 1 -- 172.21.15.153:0/2973468117 <== mon.5 v1:172.21.15.153:6791/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9bbad653-ff80-4a41-a571-666779269a6f", "id": 3}]=0 v6) ==== 112+0+2 (unknown 1082478714 0 525046306) 0x7f5e10007740 con 0x7f5e180a6c50 2024-09-13T13:51:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.729+0000 7f5e14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9bbad653-ff80-4a41-a571-666779269a6f", "id": 3}] 2024-09-13T13:51:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.729+0000 7f5e14ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:24.731 INFO:teuthology.orchestra.run.smithi153.stdout:3 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 10 monclient: shutdown 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 >> v1:172.21.15.153:6791/0 conn(0x7f5e180a6c50 legacy=0x7f5e181d6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 shutdown_connections 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 >> 172.21.15.153:0/2973468117 conn(0x7f5e180ab940 msgr2=0x7f5e180aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 shutdown_connections 2024-09-13T13:51:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.730+0000 7f5e1e8b6640 1 -- 172.21.15.153:0/2973468117 wait complete. 2024-09-13T13:51:24.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 9f1f8ec6-019a-49ae-a845-dbef0966f37d 4 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.839+0000 7f700a2db640 10 monclient: get_monmap_and_config 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.839+0000 7f700a2db640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.839+0000 7f700a2db640 10 monclient: monmap: 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:51:24.840163+0000 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:51:24.840163+0000 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.841 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.842 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 1 Processor -- start 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 1 -- start start 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient: init 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient: _add_conns ranks=[2,0,6,3,5,7,8,4,1] 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient(hunting): picked mon.noname-b con 0x7f700406c8b0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient(hunting): picked mon.noname-a con 0x7f700409bbe0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient(hunting): picked mon.noname-d con 0x7f700413f570 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70040a1080 con 0x7f700409bbe0 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700406cfe0 con 0x7f700406c8b0 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700406d4a0 con 0x7f700413f570 2024-09-13T13:51:24.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f700a2db640 10 monclient(hunting): authenticate will time out at 722.363744s 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f7003fff640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f700406c8b0 0x7f70040688e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.153:37294/0 (socket says 172.21.15.153:37294) 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f7008878640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f700413f570 0x7f7004141960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.153:40980/0 (socket says 172.21.15.153:40980) 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f70037fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f700409bbe0 0x7f7004068ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.153:50736/0 (socket says 172.21.15.153:50736) 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.840+0000 7f7003fff640 1 -- 172.21.15.153:0/1804781829 learned_addr learned my addr 172.21.15.153:0/1804781829 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7008878640 1 -- 172.21.15.153:0/1804781829 >> v1:172.21.15.121:6792/0 conn(0x7f700413f570 legacy=0x7f7004141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f70037fe640 1 -- 172.21.15.153:0/1804781829 >> v1:172.21.15.121:6789/0 conn(0x7f700409bbe0 legacy=0x7f7004068ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7003fff640 1 -- 172.21.15.153:0/1804781829 >> v1:172.21.15.121:6790/0 conn(0x7f700406c8b0 legacy=0x7f70040688e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1956445140 0 0) 0x7f700406d4a0 con 0x7f700413f570 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): my global_id is 4155 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fec003520 con 0x7f700413f570 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431858298 0 0) 0x7f6ff8001380 con 0x7f700409bbe0 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): my global_id is 4131 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f700406d4a0 con 0x7f700409bbe0 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113547814 0 0) 0x7f700406cfe0 con 0x7f700406c8b0 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient(hunting): my global_id is 4115 2024-09-13T13:51:24.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff8001380 con 0x7f700406c8b0 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1571375188 0 0) 0x7f6fec003520 con 0x7f700413f570 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 >> v1:172.21.15.121:6790/0 conn(0x7f700406c8b0 legacy=0x7f70040688e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 >> v1:172.21.15.121:6789/0 conn(0x7f700409bbe0 legacy=0x7f7004068ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 monclient: found mon.noname-d 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70040a00e0 con 0x7f700413f570 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6ffc001f20 con 0x7f700413f570 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f700a2db640 5 monclient: authenticate success, global_id 4155 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f700a2db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: dump: 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.859 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.860 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f700a2db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ffc004b60 con 0x7f700413f570 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 1 -- 172.21.15.153:0/1804781829 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6ffc005090 con 0x7f700413f570 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:24.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f7002ffd640 10 monclient: dump: 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.862 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.863 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f700a2db640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:24.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.843+0000 7f700a2db640 4 set_mon_vals no callback set 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 10 monclient: shutdown 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 1 -- 172.21.15.153:0/1804781829 >> v1:172.21.15.121:6792/0 conn(0x7f700413f570 legacy=0x7f7004141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 1 -- 172.21.15.153:0/1804781829 shutdown_connections 2024-09-13T13:51:24.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 1 -- 172.21.15.153:0/1804781829 >> 172.21.15.153:0/1804781829 conn(0x7f70040a83b0 msgr2=0x7f70040a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 1 -- 172.21.15.153:0/1804781829 shutdown_connections 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.844+0000 7f700a2db640 1 -- 172.21.15.153:0/1804781829 wait complete. 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.845+0000 7f700a2db640 10 monclient: build_initial_monmap 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.845+0000 7f700a2db640 10 monclient: monmap: 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:24.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:24.867 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:24.868 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.846+0000 7f700a2db640 1 Processor -- start 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.846+0000 7f700a2db640 1 -- start start 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.846+0000 7f700a2db640 10 monclient: init 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient: _add_conns ranks=[3,6,0,7,5,8,1,4,2] 2024-09-13T13:51:24.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient(hunting): picked mon.noname-g con 0x7f700406c8b0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient(hunting): picked mon.noname-d con 0x7f700409bbe0 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient(hunting): picked mon.noname-a con 0x7f700413f570 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7004072c40 con 0x7f700413f570 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70040a3140 con 0x7f700406c8b0 2024-09-13T13:51:24.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7004142c10 con 0x7f700409bbe0 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f700a2db640 10 monclient(hunting): authenticate will time out at 722.370238s 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f7003fff640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f700406c8b0 0x7f70041ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:44138/0 (socket says 172.21.15.153:44138) 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.847+0000 7f7003fff640 1 -- 172.21.15.153:0/1846093835 learned_addr learned my addr 172.21.15.153:0/1846093835 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7003fff640 1 -- 172.21.15.153:0/1846093835 >> v1:172.21.15.153:6790/0 conn(0x7f700406c8b0 legacy=0x7f70041ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672979923 0 0) 0x7f70040a3140 con 0x7f700406c8b0 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): my global_id is 4143 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff0003550 con 0x7f700406c8b0 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f70037fe640 1 -- 172.21.15.153:0/1846093835 >> v1:172.21.15.121:6792/0 conn(0x7f700409bbe0 legacy=0x7f70041d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7008878640 1 -- 172.21.15.153:0/1846093835 >> v1:172.21.15.121:6789/0 conn(0x7f700413f570 legacy=0x7f70041d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1271422235 0 0) 0x7f6ff0003550 con 0x7f700406c8b0 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f70040a3140 con 0x7f700406c8b0 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6ff40047e0 con 0x7f700406c8b0 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): dump: 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.874 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 521845108 0 0) 0x7f7004142c10 con 0x7f700409bbe0 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): my global_id is 4164 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ff0003550 con 0x7f700409bbe0 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116293666 0 0) 0x7f7004072c40 con 0x7f700413f570 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 10 monclient(hunting): my global_id is 4140 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.849+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7004142c10 con 0x7f700413f570 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1839035911 0 0) 0x7f70040a3140 con 0x7f700406c8b0 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 >> v1:172.21.15.121:6792/0 conn(0x7f700409bbe0 legacy=0x7f70041d33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 >> v1:172.21.15.121:6789/0 conn(0x7f700413f570 legacy=0x7f70041d6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 monclient: found mon.g 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7004072960 con 0x7f700406c8b0 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: _finish_auth 0 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:51:24.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 5 monclient: authenticate success, global_id 4143 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 10 monclient: _renew_subs 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7004074ae0 con 0x7f700406c8b0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 10 monclient: _renew_subs 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f700409e640 con 0x7f700406c8b0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ff4004cd0 con 0x7f700406c8b0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6ff4005240 con 0x7f700406c8b0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f70027fc640 4 set_mon_vals no callback set 2024-09-13T13:51:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 10 monclient: dump: 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:24.878 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6ff40058d0 con 0x7f700406c8b0 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.850+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 1875221999 0 0) 0x7f6ff40060d0 con 0x7f700406c8b0 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.851+0000 7f700a2db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.851+0000 7f700a2db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.851+0000 7f700a2db640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.851+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe4005080 con 0x7f700406c8b0 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.853+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f6ff40063f0 con 0x7f700406c8b0 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.853+0000 7f7000ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.853+0000 7f7000ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:24.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.923+0000 7f700a2db640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9f1f8ec6-019a-49ae-a845-dbef0966f37d", "id": 4}] 2024-09-13T13:51:24.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.923+0000 7f700a2db640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9f1f8ec6-019a-49ae-a845-dbef0966f37d", "id": 4}] 2024-09-13T13:51:24.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.923+0000 7f700a2db640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:24.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.923+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "osd new", "uuid": "9f1f8ec6-019a-49ae-a845-dbef0966f37d", "id": 4} v 0) -- 0x7f6fe4005370 con 0x7f700406c8b0 2024-09-13T13:51:24.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.927+0000 7f7000ff9640 1 -- 172.21.15.153:0/1846093835 <== mon.3 v1:172.21.15.153:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9f1f8ec6-019a-49ae-a845-dbef0966f37d", "id": 4}]=0 v7) ==== 112+0+2 (unknown 338522339 0 1697012583) 0x7f6ff4006bc0 con 0x7f700406c8b0 2024-09-13T13:51:24.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.928+0000 7f7000ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9f1f8ec6-019a-49ae-a845-dbef0966f37d", "id": 4}] 2024-09-13T13:51:24.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.928+0000 7f7000ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:24.929 INFO:teuthology.orchestra.run.smithi153.stdout:4 2024-09-13T13:51:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.928+0000 7f700a2db640 10 monclient: shutdown 2024-09-13T13:51:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.928+0000 7f700a2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.928+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 >> v1:172.21.15.153:6790/0 conn(0x7f700406c8b0 legacy=0x7f70041ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.929+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 shutdown_connections 2024-09-13T13:51:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.929+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 >> 172.21.15.153:0/1846093835 conn(0x7f70040a83b0 msgr2=0x7f70040a1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.929+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 shutdown_connections 2024-09-13T13:51:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:24.929+0000 7f700a2db640 1 -- 172.21.15.153:0/1846093835 wait complete. 2024-09-13T13:51:24.940 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new ba16c496-ba8f-4f30-978f-ec8c36f2656d 5 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.037+0000 7f2b5d816640 10 monclient: get_monmap_and_config 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.037+0000 7f2b5d816640 10 monclient: build_initial_monmap 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.037+0000 7f2b5d816640 10 monclient: monmap: 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:51:25.038028+0000 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:51:25.038028+0000 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:25.038 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:25.039 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.037+0000 7f2b5d816640 1 Processor -- start 2024-09-13T13:51:25.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.037+0000 7f2b5d816640 1 -- start start 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.037+0000 7f2b5d816640 10 monclient: init 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient: _add_conns ranks=[0,3,5,2,1,4,6,7,8] 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient(hunting): picked mon.noname-a con 0x7f2b580a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient(hunting): picked mon.noname-g con 0x7f2b5809d150 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient(hunting): picked mon.noname-h con 0x7f2b580a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b5809e210 con 0x7f2b580a6160 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b580a7ef0 con 0x7f2b5809d150 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b5809d970 con 0x7f2b580a6c50 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b5d816640 10 monclient(hunting): authenticate will time out at 722.561500s 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b567fc640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 0x7f2b5813e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.153:44144/0 (socket says 172.21.15.153:44144) 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b567fc640 1 -- 172.21.15.153:0/524723474 learned_addr learned my addr 172.21.15.153:0/524723474 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:25.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.038+0000 7f2b577fe640 1 --1- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6791/0 conn(0x7f2b580a6c50 0x7f2b58141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.153:42926/0 (socket says 172.21.15.153:42926) 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b567fc640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 legacy=0x7f2b5813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b567fc640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 legacy=0x7f2b5813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b567fc640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 legacy=0x7f2b5813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b567fc640 1 --1- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 0x7f2b5813e370 :-1 s=OPENED pgs=37 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b567fc640 1 --1- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 0x7f2b5813e370 :-1 s=OPENED pgs=37 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b577fe640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6791/0 conn(0x7f2b580a6c50 legacy=0x7f2b58141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6790/0 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2073434156 0 0) 0x7f2b5809d970 con 0x7f2b580a6c50 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 10 monclient(hunting): my global_id is 4136 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b40003520 con 0x7f2b580a6c50 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.040+0000 7f2b56ffd640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.121:6789/0 conn(0x7f2b580a6160 legacy=0x7f2b580a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2948114343 0 0) 0x7f2b40003520 con 0x7f2b580a6c50 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6790/0 conn(0x7f2b5809d150 legacy=0x7f2b5813e370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.121:6789/0 conn(0x7f2b580a6160 legacy=0x7f2b580a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 monclient: found mon.noname-h 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.153:6791/0 2024-09-13T13:51:25.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b5809e630 con 0x7f2b580a6c50 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 5 monclient: authenticate success, global_id 4136 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2b50001e60 con 0x7f2b580a6c50 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: dump: 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.048 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:25.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 <== mon.5 v1:172.21.15.153:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b50004b60 con 0x7f2b580a6c50 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 1 -- 172.21.15.153:0/524723474 <== mon.5 v1:172.21.15.153:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2b500050d0 con 0x7f2b580a6c50 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b55ffb640 10 monclient: dump: 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 4 set_mon_vals no callback set 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 10 monclient: shutdown 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.041+0000 7f2b5d816640 1 -- 172.21.15.153:0/524723474 >> v1:172.21.15.153:6791/0 conn(0x7f2b580a6c50 legacy=0x7f2b58141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.042+0000 7f2b5d816640 1 -- 172.21.15.153:0/524723474 shutdown_connections 2024-09-13T13:51:25.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.042+0000 7f2b5d816640 1 -- 172.21.15.153:0/524723474 >> 172.21.15.153:0/524723474 conn(0x7f2b580ab940 msgr2=0x7f2b580abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.042+0000 7f2b5d816640 1 -- 172.21.15.153:0/524723474 shutdown_connections 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.042+0000 7f2b5d816640 1 -- 172.21.15.153:0/524723474 wait complete. 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.042+0000 7f2b5d816640 10 monclient: build_initial_monmap 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.042+0000 7f2b5d816640 10 monclient: monmap: 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:25.052 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 1 Processor -- start 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 1 -- start start 2024-09-13T13:51:25.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient: init 2024-09-13T13:51:25.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:25.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient: _add_conns ranks=[7,1,8,2,5,4,6,3,0] 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient(hunting): picked mon.noname-i con 0x7f2b5809d150 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient(hunting): picked mon.noname-f con 0x7f2b580a6160 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient(hunting): picked mon.noname-e con 0x7f2b580a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b58125c70 con 0x7f2b580a6160 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b580a4600 con 0x7f2b5809d150 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b58124a50 con 0x7f2b580a6c50 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b5d816640 10 monclient(hunting): authenticate will time out at 722.566750s 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b56ffd640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 0x7f2b581393f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.153:57738/0 (socket says 172.21.15.153:57738) 2024-09-13T13:51:25.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.043+0000 7f2b56ffd640 1 -- 172.21.15.153:0/1832643371 learned_addr learned my addr 172.21.15.153:0/1832643371 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.045+0000 7f2b56ffd640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 legacy=0x7f2b581393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b567fc640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6789/0 conn(0x7f2b580a6160 legacy=0x7f2b581d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b577fe640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.121:6793/0 conn(0x7f2b580a6c50 legacy=0x7f2b581d6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159473999 0 0) 0x7f2b580a4600 con 0x7f2b5809d150 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): my global_id is 4147 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b44003520 con 0x7f2b5809d150 2024-09-13T13:51:25.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121491783 0 0) 0x7f2b58125c70 con 0x7f2b580a6160 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): my global_id is 4159 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b580a4600 con 0x7f2b580a6160 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 591903985 0 0) 0x7f2b580a4600 con 0x7f2b580a6160 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b58125c70 con 0x7f2b580a6160 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 239215431 0 0) 0x7f2b44003520 con 0x7f2b5809d150 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2b580a4600 con 0x7f2b5809d150 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2b4c001d00 con 0x7f2b580a6160 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): dump: 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:25.062 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539734982 0 0) 0x7f2b58124a50 con 0x7f2b580a6c50 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): my global_id is 4157 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b44003520 con 0x7f2b580a6c50 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b56ffd640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 legacy=0x7f2b581393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b56ffd640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 legacy=0x7f2b581393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2b48003610 con 0x7f2b5809d150 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b56ffd640 1 --1- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 0x7f2b581393f0 :-1 s=OPENED pgs=36 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b56ffd640 1 --1- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 0x7f2b581393f0 :-1 s=OPENED pgs=36 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:25.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): dump: 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.065 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6792/0 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 379775325 0 0) 0x7f2b58125c70 con 0x7f2b580a6160 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.121:6793/0 conn(0x7f2b580a6c50 legacy=0x7f2b581d6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6792/0 conn(0x7f2b5809d150 legacy=0x7f2b581393f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 monclient: found mon.f 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b58126db0 con 0x7f2b580a6160 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: _finish_auth 0 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:25.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 5 monclient: authenticate success, global_id 4159 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 10 monclient: _renew_subs 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2b581255b0 con 0x7f2b580a6160 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 10 monclient: _renew_subs 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2b58139ce0 con 0x7f2b580a6160 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2b4c004c00 con 0x7f2b580a6160 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2b4c0050f0 con 0x7f2b580a6160 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 10 monclient: dump: 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-13T13:51:25.067 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 3 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.068 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b557fa640 4 set_mon_vals no callback set 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2b4c005720 con 0x7f2b580a6160 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.046+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 2015953097 0 0) 0x7f2b4c006ff0 con 0x7f2b580a6160 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.047+0000 7f2b5d816640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.047+0000 7f2b5d816640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.047+0000 7f2b5d816640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.047+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b38005080 con 0x7f2b580a6160 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.049+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f2b4c006a60 con 0x7f2b580a6160 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.049+0000 7f2b3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:25.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.049+0000 7f2b3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:25.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.120+0000 7f2b5d816640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ba16c496-ba8f-4f30-978f-ec8c36f2656d", "id": 5}] 2024-09-13T13:51:25.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.120+0000 7f2b5d816640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ba16c496-ba8f-4f30-978f-ec8c36f2656d", "id": 5}] 2024-09-13T13:51:25.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.120+0000 7f2b5d816640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:25.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.120+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ba16c496-ba8f-4f30-978f-ec8c36f2656d", "id": 5} v 0) -- 0x7f2b38005370 con 0x7f2b580a6160 2024-09-13T13:51:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.136+0000 7f2b3ffff640 1 -- 172.21.15.153:0/1832643371 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ba16c496-ba8f-4f30-978f-ec8c36f2656d", "id": 5}]=0 v8) ==== 112+0+2 (unknown 3860772160 0 1988412176) 0x7f2b4c006e10 con 0x7f2b580a6160 2024-09-13T13:51:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.136+0000 7f2b3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ba16c496-ba8f-4f30-978f-ec8c36f2656d", "id": 5}] 2024-09-13T13:51:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.136+0000 7f2b3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:25.137 INFO:teuthology.orchestra.run.smithi153.stdout:5 2024-09-13T13:51:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 10 monclient: shutdown 2024-09-13T13:51:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 >> v1:172.21.15.153:6789/0 conn(0x7f2b580a6160 legacy=0x7f2b581d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 shutdown_connections 2024-09-13T13:51:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 >> 172.21.15.153:0/1832643371 conn(0x7f2b580ab940 msgr2=0x7f2b581c11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 shutdown_connections 2024-09-13T13:51:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-13T13:51:25.137+0000 7f2b5d816640 1 -- 172.21.15.153:0/1832643371 wait complete. 2024-09-13T13:51:25.148 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T13:51:25.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T13:51:25.152 INFO:tasks.ceph.osd.0:Started 2024-09-13T13:51:25.152 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T13:51:25.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T13:51:25.154 INFO:tasks.ceph.osd.1:Started 2024-09-13T13:51:25.154 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T13:51:25.154 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T13:51:25.157 INFO:tasks.ceph.osd.2:Started 2024-09-13T13:51:25.157 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T13:51:25.157 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T13:51:25.159 INFO:tasks.ceph.osd.3:Started 2024-09-13T13:51:25.159 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T13:51:25.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T13:51:25.161 INFO:tasks.ceph.osd.4:Started 2024-09-13T13:51:25.162 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T13:51:25.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T13:51:25.163 INFO:tasks.ceph.osd.5:Started 2024-09-13T13:51:25.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:25.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.273+0000 7fdc0c139640 10 monclient: get_monmap_and_config 2024-09-13T13:51:25.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.273+0000 7fdc0c139640 10 monclient: build_initial_monmap 2024-09-13T13:51:25.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.273+0000 7fdc0c139640 10 monclient: monmap: 2024-09-13T13:51:25.275 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:25.275 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.275 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:25.274673+0000 2024-09-13T13:51:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:25.274673+0000 2024-09-13T13:51:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:25.276 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:25.277 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.274+0000 7fdc0c139640 1 Processor -- start 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.274+0000 7fdc0c139640 1 -- start start 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.274+0000 7fdc0c139640 10 monclient: init 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient: _add_conns ranks=[2,8,3,4,7,6,0,1,5] 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient(hunting): picked mon.noname-b con 0x7fdc04138c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient(hunting): picked mon.noname-e con 0x7fdc0413bfc0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient(hunting): picked mon.noname-g con 0x7fdc0413fad0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc040a5e80 con 0x7fdc04138c50 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc09ed5640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fdc04138c50 0x7fdc040a5770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:32904/0 (socket says 172.21.15.121:32904) 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc09ed5640 1 -- 172.21.15.121:0/2422583540 learned_addr learned my addr 172.21.15.121:0/2422583540 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc096d4640 1 --1- 172.21.15.121:0/2422583540 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413bfc0 0x7fdc0413e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:35238/0 (socket says 172.21.15.121:35238) 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc04139a60 con 0x7fdc0413fad0 2024-09-13T13:51:25.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc0409df00 con 0x7fdc0413bfc0 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.275+0000 7fdc0c139640 10 monclient(hunting): authenticate will time out at 764.904585s 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc09ed5640 1 -- 172.21.15.121:0/2422583540 >> v1:172.21.15.121:6790/0 conn(0x7fdc04138c50 legacy=0x7fdc040a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc096d4640 1 -- 172.21.15.121:0/2422583540 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413bfc0 legacy=0x7fdc0413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc0a6d6640 1 -- 172.21.15.121:0/2422583540 >> v1:172.21.15.153:6790/0 conn(0x7fdc0413fad0 legacy=0x7fdc04141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147930661 0 0) 0x7fdc00001450 con 0x7fdc04138c50 2024-09-13T13:51:25.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): my global_id is 4142 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbf0003520 con 0x7fdc04138c50 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3398073276 0 0) 0x7fdc0409df00 con 0x7fdc0413bfc0 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 10 monclient(hunting): my global_id is 4175 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.277+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc00001450 con 0x7fdc0413bfc0 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696365633 0 0) 0x7fdc04139a60 con 0x7fdc0413fad0 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient(hunting): my global_id is 4170 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc0409df00 con 0x7fdc0413fad0 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4214302764 0 0) 0x7fdc00001450 con 0x7fdc0413bfc0 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 >> v1:172.21.15.153:6790/0 conn(0x7fdc0413fad0 legacy=0x7fdc04141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 >> v1:172.21.15.121:6790/0 conn(0x7fdc04138c50 legacy=0x7fdc040a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: _finish_hunting 0 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 monclient: found mon.noname-e 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0409d940 con 0x7fdc0413bfc0 2024-09-13T13:51:25.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: _finish_auth 0 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: _check_auth_tickets 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 5 monclient: authenticate success, global_id 4175 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdbf4001b80 con 0x7fdc0413bfc0 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:25.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: dump: 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.294 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.295 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdbf40047e0 con 0x7fdc0413bfc0 2024-09-13T13:51:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 4 set_mon_vals no callback set 2024-09-13T13:51:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 1 -- 172.21.15.121:0/2422583540 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdbf4004cd0 con 0x7fdc0413bfc0 2024-09-13T13:51:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc08ed3640 10 monclient: dump: 2024-09-13T13:51:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.298 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 10 monclient: shutdown 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413bfc0 legacy=0x7fdc0413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 shutdown_connections 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.278+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 >> 172.21.15.121:0/2422583540 conn(0x7fdc04069360 msgr2=0x7fdc040ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 shutdown_connections 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 1 -- 172.21.15.121:0/2422583540 wait complete. 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 10 monclient: build_initial_monmap 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 10 monclient: monmap: 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 1 Processor -- start 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 1 -- start start 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.279+0000 7fdc0c139640 10 monclient: init 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient: _add_conns ranks=[0,5,8,4,6,1,2,3,7] 2024-09-13T13:51:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient(hunting): picked mon.noname-a con 0x7fdc04138c50 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient(hunting): picked mon.noname-h con 0x7fdc0413bfc0 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc09ed5640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdc04138c50 0x7fdc040a4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:46886/0 (socket says 172.21.15.121:46886) 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc09ed5640 1 -- 172.21.15.121:0/2178589500 learned_addr learned my addr 172.21.15.121:0/2178589500 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient(hunting): picked mon.noname-e con 0x7fdc0413fad0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc04135e40 con 0x7fdc04138c50 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc040a57c0 con 0x7fdc0413bfc0 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc04138760 con 0x7fdc0413fad0 2024-09-13T13:51:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0c139640 10 monclient(hunting): authenticate will time out at 764.909602s 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0a6d6640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413fad0 legacy=0x7fdc041d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0a6d6640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413fad0 legacy=0x7fdc041d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0a6d6640 1 --1- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413fad0 0x7fdc041d6ad0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdc0a6d6640 1 --1- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413fad0 0x7fdc041d6ad0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.280+0000 7fdbfa7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6793/0 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdc09ed5640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6789/0 conn(0x7fdc04138c50 legacy=0x7fdc040a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120222644 0 0) 0x7fdc00001340 con 0x7fdc04138c50 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): my global_id is 4194 2024-09-13T13:51:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdbe8003520 con 0x7fdc04138c50 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdc096d4640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.153:6791/0 conn(0x7fdc0413bfc0 legacy=0x7fdc041d33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3363610562 0 0) 0x7fdc040a57c0 con 0x7fdc0413bfc0 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 10 monclient(hunting): my global_id is 4145 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc00001340 con 0x7fdc0413bfc0 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.282+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 30028513 0 0) 0x7fdbe8003520 con 0x7fdc04138c50 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc040a57c0 con 0x7fdc04138c50 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdc00001c30 con 0x7fdc04138c50 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 237943665 0 0) 0x7fdc040a57c0 con 0x7fdc04138c50 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6793/0 conn(0x7fdc0413fad0 legacy=0x7fdc041d6ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.153:6791/0 conn(0x7fdc0413bfc0 legacy=0x7fdc041d33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 monclient: found mon.a 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc040a91e0 con 0x7fdc04138c50 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:25.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 5 monclient: authenticate success, global_id 4194 2024-09-13T13:51:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 10 monclient: _renew_subs 2024-09-13T13:51:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdc04134fa0 con 0x7fdc04138c50 2024-09-13T13:51:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 10 monclient: _renew_subs 2024-09-13T13:51:25.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdc040a5530 con 0x7fdc04138c50 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdc00004ec0 con 0x7fdc04138c50 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdc000053b0 con 0x7fdc04138c50 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:25.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfbfff640 4 set_mon_vals no callback set 2024-09-13T13:51:25.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 10 monclient: dump: 2024-09-13T13:51:25.310 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:25.310 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:25.310 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.310 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:25.310 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:25.311 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:25.312 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:25.312 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:25.312 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:25.312 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:25.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdc000059a0 con 0x7fdc04138c50 2024-09-13T13:51:25.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.283+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 4090120631 0 0) 0x7fdc00006280 con 0x7fdc04138c50 2024-09-13T13:51:25.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.284+0000 7fdc0c139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:25.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.284+0000 7fdc0c139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:25.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.284+0000 7fdc0c139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:25.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.284+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbec005080 con 0x7fdc04138c50 2024-09-13T13:51:25.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.286+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fdc04135e40 con 0x7fdc04138c50 2024-09-13T13:51:25.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.286+0000 7fdbfa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:25.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.286+0000 7fdbfa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:25.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdc0c139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:25.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdc0c139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:25.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdc0c139640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:25.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdbec005370 con 0x7fdc04138c50 2024-09-13T13:51:25.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdbfa7fc640 1 -- 172.21.15.121:0/2178589500 <== mon.0 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 1676989776) 0x7fdc040a5e80 con 0x7fdc04138c50 2024-09-13T13:51:25.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdbfa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:25.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.344+0000 7fdbfa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:25.346 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:25.346 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":8,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:25.134237+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 10 monclient: shutdown 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 >> v1:172.21.15.121:6789/0 conn(0x7fdc04138c50 legacy=0x7fdc040a4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 shutdown_connections 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 >> 172.21.15.121:0/2178589500 conn(0x7fdc04069360 msgr2=0x7fdc04136800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 shutdown_connections 2024-09-13T13:51:25.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:25.345+0000 7fdc0c139640 1 -- 172.21.15.121:0/2178589500 wait complete. 2024-09-13T13:51:25.356 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T13:51:25.356 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T13:51:25.961 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:26.051 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.050+0000 7fce94e71640 10 monclient: get_monmap_and_config 2024-09-13T13:51:26.051 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.050+0000 7fce94e71640 10 monclient: build_initial_monmap 2024-09-13T13:51:26.051 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.050+0000 7fce94e71640 10 monclient: monmap: 2024-09-13T13:51:26.051 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-13T13:51:26.051 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:51:26.050951+0000 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:51:26.050951+0000 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:26.052 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.053 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.050+0000 7fce94e71640 1 Processor -- start 2024-09-13T13:51:26.053 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.050+0000 7fce94e71640 1 -- start start 2024-09-13T13:51:26.053 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.050+0000 7fce94e71640 10 monclient: init 2024-09-13T13:51:26.058 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient: _add_conns ranks=[6,5,7,4,3,2,8,1,0] 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient(hunting): picked mon.noname-d con 0x7fce900a1d90 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient(hunting): picked mon.noname-h con 0x7fce900a2170 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient(hunting): picked mon.noname-i con 0x7fce9013f9c0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce9009ff00 con 0x7fce900a2170 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce900a0450 con 0x7fce900a1d90 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce900a0170 con 0x7fce9013f9c0 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:26.059 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce94e71640 10 monclient(hunting): authenticate will time out at 765.680504s 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce8e59c640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fce900a1d90 0x7fce90139770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:59556/0 (socket says 172.21.15.121:59556) 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.051+0000 7fce8e59c640 1 -- 172.21.15.121:0/1578241116 learned_addr learned my addr 172.21.15.121:0/1578241116 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.053+0000 7fce8e59c640 1 -- 172.21.15.121:0/1578241116 >> v1:172.21.15.121:6792/0 conn(0x7fce900a1d90 legacy=0x7fce90139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.053+0000 7fce8ed9d640 1 -- 172.21.15.121:0/1578241116 >> v1:172.21.15.153:6792/0 conn(0x7fce9013f9c0 legacy=0x7fce90141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.053+0000 7fce8dd9b640 1 -- 172.21.15.121:0/1578241116 >> v1:172.21.15.153:6791/0 conn(0x7fce900a2170 legacy=0x7fce9013e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.053+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238424757 0 0) 0x7fce80001450 con 0x7fce900a1d90 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): my global_id is 4173 2024-09-13T13:51:26.060 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce7c003520 con 0x7fce900a1d90 2024-09-13T13:51:26.061 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867872044 0 0) 0x7fce9009ff00 con 0x7fce900a2170 2024-09-13T13:51:26.061 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:26.061 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:26.061 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:26.061 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): my global_id is 4154 2024-09-13T13:51:26.061 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce80001450 con 0x7fce900a2170 2024-09-13T13:51:26.062 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351370773 0 0) 0x7fce900a0170 con 0x7fce9013f9c0 2024-09-13T13:51:26.062 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:26.062 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:26.062 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:26.062 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient(hunting): my global_id is 4174 2024-09-13T13:51:26.062 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce9009ff00 con 0x7fce9013f9c0 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 595915702 0 0) 0x7fce7c003520 con 0x7fce900a1d90 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 >> v1:172.21.15.153:6792/0 conn(0x7fce9013f9c0 legacy=0x7fce90141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 >> v1:172.21.15.153:6791/0 conn(0x7fce900a2170 legacy=0x7fce9013e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: _finish_hunting 0 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 monclient: found mon.noname-d 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce900a0730 con 0x7fce900a1d90 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: _finish_auth 0 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: _check_auth_tickets 2024-09-13T13:51:26.063 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 5 monclient: authenticate success, global_id 4173 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fce80001cb0 con 0x7fce900a1d90 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: dump: 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.064 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce80004950 con 0x7fce900a1d90 2024-09-13T13:51:26.065 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 1 -- 172.21.15.121:0/1578241116 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fce80004e80 con 0x7fce900a1d90 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 4 set_mon_vals no callback set 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce8d59a640 10 monclient: dump: 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.066 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 10 monclient: shutdown 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 1 -- 172.21.15.121:0/1578241116 >> v1:172.21.15.121:6792/0 conn(0x7fce900a1d90 legacy=0x7fce90139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 1 -- 172.21.15.121:0/1578241116 shutdown_connections 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 1 -- 172.21.15.121:0/1578241116 >> 172.21.15.121:0/1578241116 conn(0x7fce9006e9c0 msgr2=0x7fce9006edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:26.067 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.054+0000 7fce94e71640 1 -- 172.21.15.121:0/1578241116 shutdown_connections 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.055+0000 7fce94e71640 1 -- 172.21.15.121:0/1578241116 wait complete. 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.055+0000 7fce94e71640 10 monclient: build_initial_monmap 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.055+0000 7fce94e71640 10 monclient: monmap: 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:created 0.000000 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:26.068 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 1 Processor -- start 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 1 -- start start 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient: init 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient: _add_conns ranks=[8,0,6,4,5,1,7,2,3] 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient(hunting): picked mon.noname-e con 0x7fce900a1d90 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient(hunting): picked mon.noname-a con 0x7fce900a2170 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:26.069 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient(hunting): picked mon.noname-d con 0x7fce9013f9c0 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce9013d570 con 0x7fce900a2170 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce9013ad90 con 0x7fce9013f9c0 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce900a6160 con 0x7fce900a1d90 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce94e71640 10 monclient(hunting): authenticate will time out at 765.685643s 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce8dd9b640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fce900a2170 0x7fce9013a680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:46892/0 (socket says 172.21.15.121:46892) 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce8dd9b640 1 -- 172.21.15.121:0/3628845463 learned_addr learned my addr 172.21.15.121:0/3628845463 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:26.070 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.056+0000 7fce8e59c640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fce900a1d90 0x7fce90139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:35258/0 (socket says 172.21.15.121:35258) 2024-09-13T13:51:26.076 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.058+0000 7fce8ed9d640 1 -- 172.21.15.121:0/3628845463 >> v1:172.21.15.121:6792/0 conn(0x7fce9013f9c0 legacy=0x7fce901d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:26.076 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.058+0000 7fce8e59c640 1 -- 172.21.15.121:0/3628845463 >> v1:172.21.15.121:6793/0 conn(0x7fce900a1d90 legacy=0x7fce90139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:26.076 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.058+0000 7fce8dd9b640 1 -- 172.21.15.121:0/3628845463 >> v1:172.21.15.121:6789/0 conn(0x7fce900a2170 legacy=0x7fce9013a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:26.076 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538503427 0 0) 0x7fce9013ad90 con 0x7fce9013f9c0 2024-09-13T13:51:26.076 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): my global_id is 4182 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce70003520 con 0x7fce9013f9c0 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1438876559 0 0) 0x7fce900a6160 con 0x7fce900a1d90 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): my global_id is 4184 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce9013ad90 con 0x7fce900a1d90 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081237136 0 0) 0x7fce9013d570 con 0x7fce900a2170 2024-09-13T13:51:26.077 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): my global_id is 4203 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce900a6160 con 0x7fce900a2170 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3712493705 0 0) 0x7fce9013ad90 con 0x7fce900a1d90 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce9013d570 con 0x7fce900a1d90 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fce80004e40 con 0x7fce900a1d90 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.078 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:26.079 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1272449015 0 0) 0x7fce70003520 con 0x7fce9013f9c0 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce9013ad90 con 0x7fce9013f9c0 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 451786236 0 0) 0x7fce900a6160 con 0x7fce900a2170 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fce70003520 con 0x7fce900a2170 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fce88003990 con 0x7fce9013f9c0 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:26.080 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fce78001d00 con 0x7fce900a2170 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:26.081 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:26.082 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1504772117 0 0) 0x7fce9013d570 con 0x7fce900a1d90 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 >> v1:172.21.15.121:6792/0 conn(0x7fce9013f9c0 legacy=0x7fce901d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 >> v1:172.21.15.121:6789/0 conn(0x7fce900a2170 legacy=0x7fce9013a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 monclient: found mon.e 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce90142e80 con 0x7fce900a1d90 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: _finish_auth 0 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:26.083 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 5 monclient: authenticate success, global_id 4184 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 10 monclient: _renew_subs 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fce900a6b00 con 0x7fce900a1d90 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 10 monclient: _renew_subs 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fce9013d290 con 0x7fce900a1d90 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fce80001cd0 con 0x7fce900a1d90 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fce80001750 con 0x7fce900a1d90 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:26.084 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce87fff640 4 set_mon_vals no callback set 2024-09-13T13:51:26.085 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 10 monclient: dump: 2024-09-13T13:51:26.085 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:26.085 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:26.085 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.088 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:26.088 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:26.088 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:26.089 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fce80005960 con 0x7fce900a1d90 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.059+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 4090120631 0 0) 0x7fce80007050 con 0x7fce900a1d90 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.060+0000 7fce94e71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.060+0000 7fce94e71640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.060+0000 7fce94e71640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.060+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce74005080 con 0x7fce900a1d90 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.062+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fce900a0450 con 0x7fce900a1d90 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.062+0000 7fce86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:26.090 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.062+0000 7fce86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:26.124 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.123+0000 7fce94e71640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.123+0000 7fce94e71640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.123+0000 7fce94e71640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.123+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce74005370 con 0x7fce900a1d90 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce86ffd640 1 -- 172.21.15.121:0/3628845463 <== mon.8 v1:172.21.15.121:6793/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 1676989776) 0x7fce80006b70 con 0x7fce900a1d90 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stdout: 2024-09-13T13:51:26.125 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":8,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:25.134237+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:26.126 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 10 monclient: shutdown 2024-09-13T13:51:26.126 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:26.126 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 >> v1:172.21.15.121:6793/0 conn(0x7fce900a1d90 legacy=0x7fce90139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:26.126 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 shutdown_connections 2024-09-13T13:51:26.127 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 >> 172.21.15.121:0/3628845463 conn(0x7fce9006e9c0 msgr2=0x7fce9013f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:26.127 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 shutdown_connections 2024-09-13T13:51:26.127 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:26.124+0000 7fce94e71640 1 -- 172.21.15.121:0/3628845463 wait complete. 2024-09-13T13:51:26.134 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-13T13:51:26.371 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-09-13T13:51:26.370+0000 7fed5f231640 -1 Falling back to public interface 2024-09-13T13:51:26.615 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-13T13:51:26.614+0000 7fb65b81e640 -1 Falling back to public interface 2024-09-13T13:51:26.617 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-13T13:51:26.616+0000 7fdc99415640 -1 Falling back to public interface 2024-09-13T13:51:26.631 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-09-13T13:51:26.630+0000 7ff9cebd4640 -1 Falling back to public interface 2024-09-13T13:51:26.635 INFO:tasks.ceph.osd.3.smithi153.stderr:2024-09-13T13:51:26.634+0000 7f8ea3ada640 -1 Falling back to public interface 2024-09-13T13:51:26.853 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-13T13:51:26.852+0000 7f1a3220d640 -1 Falling back to public interface 2024-09-13T13:51:28.234 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.232+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T13:51:28.234 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.233+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-09-13T13:51:28.234 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.233+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-09-13T13:51:28.240 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.239+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T13:51:28.240 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.239+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-09-13T13:51:28.240 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.239+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-09-13T13:51:28.240 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.239+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-09-13T13:51:28.731 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-09-13T13:51:28.730+0000 7f03778a7640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-09-13T13:51:29.961 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-09-13T13:51:29.960+0000 7fed5f231640 -1 osd.4 0 log_to_monitors true 2024-09-13T13:51:30.146 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-13T13:51:30.145+0000 7fdc99415640 -1 osd.2 0 log_to_monitors true 2024-09-13T13:51:30.219 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-09-13T13:51:30.218+0000 7ff9cebd4640 -1 osd.5 0 log_to_monitors true 2024-09-13T13:51:30.447 INFO:tasks.ceph.osd.3.smithi153.stderr:2024-09-13T13:51:30.446+0000 7f8ea3ada640 -1 osd.3 0 log_to_monitors true 2024-09-13T13:51:30.662 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-13T13:51:30.661+0000 7fb65b81e640 -1 osd.1 0 log_to_monitors true 2024-09-13T13:51:30.835 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-13T13:51:30.834+0000 7f1a3220d640 -1 osd.0 0 log_to_monitors true 2024-09-13T13:51:31.396 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-13T13:51:31.395+0000 7fdc94cf5640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T13:51:31.399 INFO:tasks.ceph.osd.2.smithi121.stderr:2024-09-13T13:51:31.399+0000 7fdc90b1c640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:51:32.398 INFO:tasks.ceph.osd.3.smithi153.stderr:2024-09-13T13:51:32.397+0000 7f8e9fa69640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T13:51:32.403 INFO:tasks.ceph.osd.3.smithi153.stderr:2024-09-13T13:51:32.403+0000 7f8e9b890640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:51:32.404 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-13T13:51:32.404+0000 7f1a2dc3a640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T13:51:32.408 INFO:tasks.ceph.osd.0.smithi121.stderr:2024-09-13T13:51:32.407+0000 7f1a29a61640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:51:32.739 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:32.830 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.828+0000 7f4810211640 10 monclient: get_monmap_and_config 2024-09-13T13:51:32.830 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.828+0000 7f4810211640 10 monclient: build_initial_monmap 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.828+0000 7f4810211640 10 monclient: monmap: 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:51:32.830525+0000 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:51:32.830525+0000 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:32.831 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:32.832 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:32.832 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:32.832 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:32.832 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.833 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.829+0000 7f4810211640 1 Processor -- start 2024-09-13T13:51:32.833 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.829+0000 7f4810211640 1 -- start start 2024-09-13T13:51:32.833 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.829+0000 7f4810211640 10 monclient: init 2024-09-13T13:51:32.833 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:32.833 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient: _add_conns ranks=[2,5,7,4,0,3,8,6,1] 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient(hunting): picked mon.noname-b con 0x7f48080a1d90 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient(hunting): picked mon.noname-h con 0x7f48080a2170 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient(hunting): picked mon.noname-i con 0x7f480813f9c0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480809ff00 con 0x7f48080a1d90 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48080a0450 con 0x7f48080a2170 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48080a0170 con 0x7f480813f9c0 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f4810211640 10 monclient(hunting): authenticate will time out at 772.460164s 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f480dfad640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f48080a1d90 0x7f4808139770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:32932/0 (socket says 172.21.15.121:32932) 2024-09-13T13:51:32.834 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.830+0000 7f480dfad640 1 -- 172.21.15.121:0/1610540622 learned_addr learned my addr 172.21.15.121:0/1610540622 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480dfad640 1 -- 172.21.15.121:0/1610540622 >> v1:172.21.15.121:6790/0 conn(0x7f48080a1d90 legacy=0x7f4808139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480d7ac640 1 -- 172.21.15.121:0/1610540622 >> v1:172.21.15.153:6791/0 conn(0x7f48080a2170 legacy=0x7f480813e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480e7ae640 1 -- 172.21.15.121:0/1610540622 >> v1:172.21.15.153:6792/0 conn(0x7f480813f9c0 legacy=0x7f4808141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96190396 0 0) 0x7f480809ff00 con 0x7f48080a1d90 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient(hunting): _negotiate 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient(hunting): my global_id is 4178 2024-09-13T13:51:32.838 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47f4003520 con 0x7f48080a1d90 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513413687 0 0) 0x7f47f4003520 con 0x7f48080a1d90 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 >> v1:172.21.15.153:6792/0 conn(0x7f480813f9c0 legacy=0x7f4808141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 >> v1:172.21.15.153:6791/0 conn(0x7f48080a2170 legacy=0x7f480813e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: _finish_hunting 0 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 monclient: found mon.noname-b 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48080a0730 con 0x7f48080a1d90 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: _finish_auth 0 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: _check_auth_tickets 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f4810211640 5 monclient: authenticate success, global_id 4178 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4804001ae0 con 0x7f48080a1d90 2024-09-13T13:51:32.839 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f4810211640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f480cfab640 10 monclient: dump: 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:32.840 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:32.841 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.832+0000 7f4810211640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f48040048a0 con 0x7f48080a1d90 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f480cfab640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f480cfab640 1 -- 172.21.15.121:0/1610540622 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4804004dd0 con 0x7f48080a1d90 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 4 set_mon_vals no callback set 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f480cfab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f480cfab640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f480cfab640 10 monclient: dump: 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.842 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 10 monclient: shutdown 2024-09-13T13:51:32.843 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 1 -- 172.21.15.121:0/1610540622 >> v1:172.21.15.121:6790/0 conn(0x7f48080a1d90 legacy=0x7f4808139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 1 -- 172.21.15.121:0/1610540622 shutdown_connections 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 1 -- 172.21.15.121:0/1610540622 >> 172.21.15.121:0/1610540622 conn(0x7f480806e9c0 msgr2=0x7f480806edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 1 -- 172.21.15.121:0/1610540622 shutdown_connections 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.833+0000 7f4810211640 1 -- 172.21.15.121:0/1610540622 wait complete. 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.834+0000 7f4810211640 10 monclient: build_initial_monmap 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.834+0000 7f4810211640 10 monclient: monmap: 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:created 0.000000 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:32.844 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.834+0000 7f4810211640 1 Processor -- start 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.834+0000 7f4810211640 1 -- start start 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.834+0000 7f4810211640 10 monclient: init 2024-09-13T13:51:32.845 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.834+0000 7f4810211640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 10 monclient: _add_conns ranks=[2,5,0,7,8,6,3,4,1] 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 10 monclient(hunting): picked mon.noname-b con 0x7f48080a1d90 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 10 monclient(hunting): picked mon.noname-h con 0x7f48080a2170 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 10 monclient(hunting): picked mon.noname-a con 0x7f480813f9c0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480813d5c0 con 0x7f480813f9c0 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480813ad90 con 0x7f48080a1d90 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48080a9530 con 0x7f48080a2170 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f4810211640 10 monclient(hunting): authenticate will time out at 772.465148s 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f480dfad640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f48080a1d90 0x7f4808139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:32934/0 (socket says 172.21.15.121:32934) 2024-09-13T13:51:32.846 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.835+0000 7f480dfad640 1 -- 172.21.15.121:0/655545089 learned_addr learned my addr 172.21.15.121:0/655545089 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:32.850 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f480e7ae640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.121:6789/0 conn(0x7f480813f9c0 legacy=0x7f48081d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:32.850 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f480dfad640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.121:6790/0 conn(0x7f48080a1d90 legacy=0x7f4808139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:32.850 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f480d7ac640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.153:6791/0 conn(0x7f48080a2170 legacy=0x7f480813a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:32.850 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4027601651 0 0) 0x7f480813d5c0 con 0x7f480813f9c0 2024-09-13T13:51:32.850 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:32.850 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): my global_id is 4239 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47ec003520 con 0x7f480813f9c0 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259236467 0 0) 0x7f480813ad90 con 0x7f48080a1d90 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): my global_id is 4187 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480813d5c0 con 0x7f48080a1d90 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2267878815 0 0) 0x7f47ec003520 con 0x7f480813f9c0 2024-09-13T13:51:32.851 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f480813ad90 con 0x7f480813f9c0 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f48000020a0 con 0x7f480813f9c0 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:32.852 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2675423815 0 0) 0x7f480813d5c0 con 0x7f48080a1d90 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f47ec003520 con 0x7f48080a1d90 2024-09-13T13:51:32.853 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4804001d30 con 0x7f48080a1d90 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:32.854 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 858078096 0 0) 0x7f480813ad90 con 0x7f480813f9c0 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.153:6791/0 conn(0x7f48080a2170 legacy=0x7f480813a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.121:6790/0 conn(0x7f48080a1d90 legacy=0x7f4808139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:32.855 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 monclient: found mon.a 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4808142dd0 con 0x7f480813f9c0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f47fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f4810211640 5 monclient: authenticate success, global_id 4239 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f4810211640 10 monclient: _renew_subs 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f4810211640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.837+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f480813d2e0 con 0x7f480813f9c0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f4810211640 10 monclient: _renew_subs 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f4810211640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f48080a5380 con 0x7f480813f9c0 2024-09-13T13:51:32.856 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4800003c40 con 0x7f480813f9c0 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f48000041b0 con 0x7f480813f9c0 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fffff640 4 set_mon_vals no callback set 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 10 monclient: dump: 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:32.857 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.838+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 4) ==== 50328+0+0 (unknown 680596143 0 0) 0x7f4800004350 con 0x7f480813f9c0 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.839+0000 7f4810211640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.839+0000 7f4810211640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.839+0000 7f4810211640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:32.858 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.839+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47f0005080 con 0x7f480813f9c0 2024-09-13T13:51:32.859 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.839+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(11..11 src has 1..11) ==== 2067+0+0 (unknown 4103646402 0 0) 0x7f480004ddb0 con 0x7f480813f9c0 2024-09-13T13:51:32.859 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.839+0000 7f480dfad640 10 monclient: get_auth_request con 0x7f47ec040240 auth_method 0 2024-09-13T13:51:32.859 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.843+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f4800017aa0 con 0x7f480813f9c0 2024-09-13T13:51:32.859 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.843+0000 7f47fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:32.859 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:32.843+0000 7f47fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:33.034 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f4810211640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:33.034 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f4810211640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:33.034 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f4810211640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:33.034 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47f0005370 con 0x7f480813f9c0 2024-09-13T13:51:33.035 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f47fe7fc640 1 -- 172.21.15.121:0/655545089 <== mon.0 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+5871 (unknown 238102980 0 2536802107) 0x7f48000205a0 con 0x7f480813f9c0 2024-09-13T13:51:33.035 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f47fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:33.035 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.032+0000 7f47fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:33.035 INFO:teuthology.misc.health.smithi121.stdout: 2024-09-13T13:51:33.035 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":11,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:32.390301+0000","last_up_change":"2024-09-13T13:51:32.390301+0000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2141781943}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2141781943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2141781943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2141781943}]},"public_addr":"172.21.15.121:6804/2141781943","cluster_addr":"172.21.15.121:6805/2141781943","heartbeat_back_addr":"172.21.15.121:6807/2141781943","heartbeat_front_addr":"172.21.15.121:6806/2141781943","state":["exists","up"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:33.036 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.034+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.153:6812/1206688214 conn(0x7f47ec040240 legacy=0x7f47ec042700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.034+0000 7f4810211640 10 monclient: shutdown 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.034+0000 7f4810211640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.034+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 >> v1:172.21.15.121:6789/0 conn(0x7f480813f9c0 legacy=0x7f48081d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.034+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 shutdown_connections 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.035+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 >> 172.21.15.121:0/655545089 conn(0x7f480806e9c0 msgr2=0x7f480813f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.035+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 shutdown_connections 2024-09-13T13:51:33.037 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:33.035+0000 7f4810211640 1 -- 172.21.15.121:0/655545089 wait complete. 2024-09-13T13:51:33.045 DEBUG:teuthology.misc:1 of 6 OSDs are up 2024-09-13T13:51:34.248 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-09-13T13:51:34.247+0000 7ff9cab63640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T13:51:34.249 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-09-13T13:51:34.249+0000 7fed5a785640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T13:51:34.253 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-09-13T13:51:34.252+0000 7fed525a4640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:51:34.254 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-09-13T13:51:34.253+0000 7ff9c2982640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:51:35.039 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-13T13:51:35.037+0000 7fb657111640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T13:51:35.042 INFO:tasks.ceph.osd.1.smithi121.stderr:2024-09-13T13:51:35.040+0000 7fb64ef30640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T13:51:39.651 DEBUG:teuthology.orchestra.run.smithi121:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:39.749 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.747+0000 7f6f9e260640 10 monclient: get_monmap_and_config 2024-09-13T13:51:39.749 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.747+0000 7f6f9e260640 10 monclient: build_initial_monmap 2024-09-13T13:51:39.749 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.747+0000 7f6f9e260640 10 monclient: monmap: 2024-09-13T13:51:39.749 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-13T13:51:39.749 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:51:39.749060+0000 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:51:39.749060+0000 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:39.750 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:39.751 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 1 Processor -- start 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 1 -- start start 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient: init 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient: _add_conns ranks=[4,8,0,5,2,7,3,6,1] 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient(hunting): picked mon.noname-c con 0x7f6f98138f00 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient(hunting): picked mon.noname-e con 0x7f6f981399f0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient(hunting): picked mon.noname-a con 0x7f6f9813fae0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9809d110 con 0x7f6f9813fae0 2024-09-13T13:51:39.752 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f980a7150 con 0x7f6f98138f00 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9809e170 con 0x7f6f981399f0 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f9e260640 10 monclient(hunting): authenticate will time out at 779.378809s 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f977fe640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f6f98138f00 0x7f6f981392e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:57096/0 (socket says 172.21.15.121:57096) 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f977fe640 1 -- 172.21.15.121:0/2806142916 learned_addr learned my addr 172.21.15.121:0/2806142916 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f97fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6f9813fae0 0x7f6f98141ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58002/0 (socket says 172.21.15.121:58002) 2024-09-13T13:51:39.753 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.748+0000 7f6f96ffd640 1 --1- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6793/0 conn(0x7f6f981399f0 0x7f6f9813e3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:48940/0 (socket says 172.21.15.121:48940) 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f977fe640 1 -- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6791/0 conn(0x7f6f98138f00 legacy=0x7f6f981392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f97fff640 1 -- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6789/0 conn(0x7f6f9813fae0 legacy=0x7f6f98141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f96ffd640 1 -- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6793/0 conn(0x7f6f981399f0 legacy=0x7f6f9813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375622425 0 0) 0x7f6f980a7150 con 0x7f6f98138f00 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): my global_id is 4180 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f84003520 con 0x7f6f98138f00 2024-09-13T13:51:39.757 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1350622425 0 0) 0x7f6f9809d110 con 0x7f6f9813fae0 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): my global_id is 4257 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f980a7150 con 0x7f6f9813fae0 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580070815 0 0) 0x7f6f9809e170 con 0x7f6f981399f0 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient(hunting): my global_id is 4238 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f9809d110 con 0x7f6f981399f0 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 633663039 0 0) 0x7f6f84003520 con 0x7f6f98138f00 2024-09-13T13:51:39.758 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6793/0 conn(0x7f6f981399f0 legacy=0x7f6f9813e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6789/0 conn(0x7f6f9813fae0 legacy=0x7f6f98141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 monclient: found mon.noname-c 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9809c9c0 con 0x7f6f98138f00 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6f88001ae0 con 0x7f6f98138f00 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f9e260640 5 monclient: authenticate success, global_id 4180 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:39.759 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: dump: 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:39.760 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f9e260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f880048a0 con 0x7f6f98138f00 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 1 -- 172.21.15.121:0/2806142916 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6f88004e10 con 0x7f6f98138f00 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f967fc640 10 monclient: dump: 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:39.761 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.762 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f9e260640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.751+0000 7f6f9e260640 4 set_mon_vals no callback set 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 10 monclient: shutdown 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 1 -- 172.21.15.121:0/2806142916 >> v1:172.21.15.121:6791/0 conn(0x7f6f98138f00 legacy=0x7f6f981392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 1 -- 172.21.15.121:0/2806142916 shutdown_connections 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 1 -- 172.21.15.121:0/2806142916 >> 172.21.15.121:0/2806142916 conn(0x7f6f980a8240 msgr2=0x7f6f980a8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 1 -- 172.21.15.121:0/2806142916 shutdown_connections 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 1 -- 172.21.15.121:0/2806142916 wait complete. 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 10 monclient: build_initial_monmap 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.752+0000 7f6f9e260640 10 monclient: monmap: 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:epoch 0 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:39.763 INFO:teuthology.misc.health.smithi121.stderr:created 0.000000 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 1 Processor -- start 2024-09-13T13:51:39.764 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 1 -- start start 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 10 monclient: init 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 10 monclient: _add_conns ranks=[1,0,2,3,5,6,4,8,7] 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 10 monclient(hunting): picked mon.noname-f con 0x7f6f98138f00 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 10 monclient(hunting): picked mon.noname-a con 0x7f6f981399f0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 10 monclient(hunting): picked mon.noname-b con 0x7f6f9813fae0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f980a2380 con 0x7f6f981399f0 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9806f180 con 0x7f6f98138f00 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.753+0000 7f6f9e260640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9813bdd0 con 0x7f6f9813fae0 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.754+0000 7f6f9e260640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.754+0000 7f6f9e260640 10 monclient(hunting): authenticate will time out at 779.384011s 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.754+0000 7f6f96ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6f981399f0 0x7f6f9813a8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58014/0 (socket says 172.21.15.121:58014) 2024-09-13T13:51:39.765 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.754+0000 7f6f96ffd640 1 -- 172.21.15.121:0/3561127944 learned_addr learned my addr 172.21.15.121:0/3561127944 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f96ffd640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.121:6789/0 conn(0x7f6f981399f0 legacy=0x7f6f9813a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f97fff640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.121:6790/0 conn(0x7f6f9813fae0 legacy=0x7f6f981d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f977fe640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.153:6789/0 conn(0x7f6f98138f00 legacy=0x7f6f9813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510451596 0 0) 0x7f6f9813bdd0 con 0x7f6f9813fae0 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): my global_id is 4205 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f78003520 con 0x7f6f9813fae0 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 38258741 0 0) 0x7f6f980a2380 con 0x7f6f981399f0 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:39.769 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): my global_id is 4266 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f9813bdd0 con 0x7f6f981399f0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1200320419 0 0) 0x7f6f78003520 con 0x7f6f9813fae0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f980a2380 con 0x7f6f9813fae0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 991481129 0 0) 0x7f6f9813bdd0 con 0x7f6f981399f0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6f78003520 con 0x7f6f981399f0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6f90002090 con 0x7f6f9813fae0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): dump: 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.770 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:39.771 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2615746178 0 0) 0x7f6f9806f180 con 0x7f6f98138f00 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): my global_id is 4240 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f9813bdd0 con 0x7f6f98138f00 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6f800047e0 con 0x7f6f981399f0 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.756+0000 7f6f8ffff640 10 monclient(hunting): dump: 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.772 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1305754851 0 0) 0x7f6f980a2380 con 0x7f6f9813fae0 2024-09-13T13:51:39.773 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.153:6789/0 conn(0x7f6f98138f00 legacy=0x7f6f9813a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.121:6789/0 conn(0x7f6f981399f0 legacy=0x7f6f9813a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 monclient: found mon.b 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9813ce30 con 0x7f6f9813fae0 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: _finish_auth 0 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 5 monclient: authenticate success, global_id 4205 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 10 monclient: _renew_subs 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9813d110 con 0x7f6f9813fae0 2024-09-13T13:51:39.774 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 10 monclient: _renew_subs 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6f9813b910 con 0x7f6f9813fae0 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6f90004e10 con 0x7f6f9813fae0 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6f90005340 con 0x7f6f9813fae0 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f95ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.757+0000 7f6f8ffff640 10 monclient: dump: 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:epoch 1 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:39.775 INFO:teuthology.misc.health.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr: 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f6f900055e0 con 0x7f6f9813fae0 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f9e260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:39.776 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f9e260640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f9e260640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7c005080 con 0x7f6f9813fae0 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 3153+0+0 (unknown 39099219 0 0) 0x7f6f9004e280 con 0x7f6f9813fae0 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.758+0000 7f6f977fe640 10 monclient: get_auth_request con 0x7f6f78040130 auth_method 0 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.763+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f6f90017930 con 0x7f6f9813fae0 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.763+0000 7f6f8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:39.777 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.763+0000 7f6f8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:39.940 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f9e260640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:39.940 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f9e260640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:39.940 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f9e260640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:39.940 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f7c005370 con 0x7f6f9813fae0 2024-09-13T13:51:39.940 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f8ffff640 1 -- 172.21.15.121:0/3561127944 <== mon.2 v1:172.21.15.121:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+9073 (unknown 1054498076 0 2658205831) 0x7f6f9001baa0 con 0x7f6f9813fae0 2024-09-13T13:51:39.940 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:39.941 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.938+0000 7f6f8ffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:39.941 INFO:teuthology.misc.health.smithi121.stdout: 2024-09-13T13:51:39.941 INFO:teuthology.misc.health.smithi121.stdout:{"epoch":16,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:38.378964+0000","last_up_change":"2024-09-13T13:51:36.267919+0000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:51:35.278046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":3060725185}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":3060725185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":3060725185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":3060725185}]},"public_addr":"172.21.15.121:6808/3060725185","cluster_addr":"172.21.15.121:6809/3060725185","heartbeat_back_addr":"172.21.15.121:6811/3060725185","heartbeat_front_addr":"172.21.15.121:6810/3060725185","state":["exists","up"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":1651307734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1651307734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1651307734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1651307734}]},"public_addr":"172.21.15.121:6800/1651307734","cluster_addr":"172.21.15.121:6801/1651307734","heartbeat_back_addr":"172.21.15.121:6803/1651307734","heartbeat_front_addr":"172.21.15.121:6802/1651307734","state":["exists","up"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2141781943}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2141781943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2141781943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2141781943}]},"public_addr":"172.21.15.121:6804/2141781943","cluster_addr":"172.21.15.121:6805/2141781943","heartbeat_back_addr":"172.21.15.121:6807/2141781943","heartbeat_front_addr":"172.21.15.121:6806/2141781943","state":["exists","up"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6808","nonce":934815702}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6809","nonce":934815702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6811","nonce":934815702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6810","nonce":934815702}]},"public_addr":"172.21.15.153:6808/934815702","cluster_addr":"172.21.15.153:6809/934815702","heartbeat_back_addr":"172.21.15.153:6811/934815702","heartbeat_front_addr":"172.21.15.153:6810/934815702","state":["exists","up"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6800","nonce":1083052486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6801","nonce":1083052486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6803","nonce":1083052486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6802","nonce":1083052486}]},"public_addr":"172.21.15.153:6800/1083052486","cluster_addr":"172.21.15.153:6801/1083052486","heartbeat_back_addr":"172.21.15.153:6803/1083052486","heartbeat_front_addr":"172.21.15.153:6802/1083052486","state":["exists","up"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6804","nonce":3215471673}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6805","nonce":3215471673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6807","nonce":3215471673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6806","nonce":3215471673}]},"public_addr":"172.21.15.153:6804/3215471673","cluster_addr":"172.21.15.153:6805/3215471673","heartbeat_back_addr":"172.21.15.153:6807/3215471673","heartbeat_front_addr":"172.21.15.153:6806/3215471673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.867807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.618229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.154306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.400091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:30.983001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.211095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.940+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.153:6812/1206688214 conn(0x7f6f78040130 legacy=0x7f6f780425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.940+0000 7f6f9e260640 10 monclient: shutdown 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.941+0000 7f6f9e260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.941+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 >> v1:172.21.15.121:6790/0 conn(0x7f6f9813fae0 legacy=0x7f6f981d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.941+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 shutdown_connections 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.941+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 >> 172.21.15.121:0/3561127944 conn(0x7f6f980a8240 msgr2=0x7f6f980aa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.941+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 shutdown_connections 2024-09-13T13:51:39.943 INFO:teuthology.misc.health.smithi121.stderr:2024-09-13T13:51:39.941+0000 7f6f9e260640 1 -- 172.21.15.121:0/3561127944 wait complete. 2024-09-13T13:51:39.952 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-13T13:51:39.952 INFO:tasks.ceph:Creating RBD pool 2024-09-13T13:51:39.952 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-13T13:51:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.046+0000 7fbb7dca9640 10 monclient: get_monmap_and_config 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.046+0000 7fbb7dca9640 10 monclient: build_initial_monmap 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.046+0000 7fbb7dca9640 10 monclient: monmap: 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:40.047935+0000 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:40.047935+0000 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:40.049 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.046+0000 7fbb7dca9640 1 Processor -- start 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.046+0000 7fbb7dca9640 1 -- start start 2024-09-13T13:51:40.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.046+0000 7fbb7dca9640 10 monclient: init 2024-09-13T13:51:40.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient: _add_conns ranks=[6,0,4,5,1,8,7,3,2] 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient(hunting): picked mon.noname-d con 0x7fbb780a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient(hunting): picked mon.noname-a con 0x7fbb7809d150 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient(hunting): picked mon.noname-c con 0x7fbb780a6c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7809e210 con 0x7fbb7809d150 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb780a7ef0 con 0x7fbb780a6c50 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7809d970 con 0x7fbb780a6160 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb7dca9640 10 monclient(hunting): authenticate will time out at 779.677495s 2024-09-13T13:51:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb76ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbb7809d150 0x7fbb7813e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:58028/0 (socket says 172.21.15.121:58028) 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.047+0000 7fbb76ffd640 1 -- 172.21.15.121:0/2294404391 learned_addr learned my addr 172.21.15.121:0/2294404391 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb76ffd640 1 -- 172.21.15.121:0/2294404391 >> v1:172.21.15.121:6789/0 conn(0x7fbb7809d150 legacy=0x7fbb7813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb777fe640 1 -- 172.21.15.121:0/2294404391 >> v1:172.21.15.121:6792/0 conn(0x7fbb780a6160 legacy=0x7fbb780a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb77fff640 1 -- 172.21.15.121:0/2294404391 >> v1:172.21.15.121:6791/0 conn(0x7fbb780a6c50 legacy=0x7fbb78141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4263171410 0 0) 0x7fbb7809d970 con 0x7fbb780a6160 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb767fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb767fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.049+0000 7fbb767fc640 10 monclient(hunting): my global_id is 4227 2024-09-13T13:51:40.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb60003520 con 0x7fbb780a6160 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584375933 0 0) 0x7fbb7809e210 con 0x7fbb7809d150 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient(hunting): my global_id is 4275 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb7809d970 con 0x7fbb7809d150 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1466396779 0 0) 0x7fbb60003520 con 0x7fbb780a6160 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 >> v1:172.21.15.121:6791/0 conn(0x7fbb780a6c50 legacy=0x7fbb78141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 >> v1:172.21.15.121:6789/0 conn(0x7fbb7809d150 legacy=0x7fbb7813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 monclient: found mon.noname-d 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:40.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7809e630 con 0x7fbb780a6160 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 5 monclient: authenticate success, global_id 4227 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbb68001a60 con 0x7fbb780a6160 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: dump: 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.058 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb68001d30 con 0x7fbb780a6160 2024-09-13T13:51:40.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 1 -- 172.21.15.121:0/2294404391 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbb68003950 con 0x7fbb780a6160 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 4 set_mon_vals no callback set 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb767fc640 10 monclient: dump: 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 10 monclient: shutdown 2024-09-13T13:51:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 1 -- 172.21.15.121:0/2294404391 >> v1:172.21.15.121:6792/0 conn(0x7fbb780a6160 legacy=0x7fbb780a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 1 -- 172.21.15.121:0/2294404391 shutdown_connections 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.050+0000 7fbb7dca9640 1 -- 172.21.15.121:0/2294404391 >> 172.21.15.121:0/2294404391 conn(0x7fbb780ab940 msgr2=0x7fbb780abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.051+0000 7fbb7dca9640 1 -- 172.21.15.121:0/2294404391 shutdown_connections 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.051+0000 7fbb7dca9640 1 -- 172.21.15.121:0/2294404391 wait complete. 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.051+0000 7fbb7dca9640 10 monclient: build_initial_monmap 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.051+0000 7fbb7dca9640 10 monclient: monmap: 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:40.062 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.051+0000 7fbb7dca9640 1 Processor -- start 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 1 -- start start 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient: init 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:40.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient: _add_conns ranks=[4,8,1,3,0,6,5,7,2] 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient(hunting): picked mon.noname-c con 0x7fbb7809d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient(hunting): picked mon.noname-e con 0x7fbb780a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient(hunting): picked mon.noname-f con 0x7fbb780a6c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb780a46a0 con 0x7fbb780a6c50 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb781247f0 con 0x7fbb7809d150 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb78126b50 con 0x7fbb780a6160 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb7dca9640 10 monclient(hunting): authenticate will time out at 779.682610s 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb777fe640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fbb7809d150 0x7fbb78139190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:57112/0 (socket says 172.21.15.121:57112) 2024-09-13T13:51:40.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.052+0000 7fbb777fe640 1 -- 172.21.15.121:0/1708528230 learned_addr learned my addr 172.21.15.121:0/1708528230 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.054+0000 7fbb777fe640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.121:6791/0 conn(0x7fbb7809d150 legacy=0x7fbb78139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.054+0000 7fbb76ffd640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.121:6793/0 conn(0x7fbb780a6160 legacy=0x7fbb781d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.054+0000 7fbb777fe640 1 --1- 172.21.15.121:0/1708528230 >> v1:172.21.15.121:6791/0 conn(0x7fbb7809d150 0x7fbb78139190 :-1 s=OPENED pgs=46 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7fbb781247f0 auth_reply(proto 2 0 (0) Success) 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb77fff640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.153:6789/0 conn(0x7fbb780a6c50 legacy=0x7fbb781d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800680934 0 0) 0x7fbb780a46a0 con 0x7fbb780a6c50 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): my global_id is 4249 2024-09-13T13:51:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb64003520 con 0x7fbb780a6c50 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2231356376 0 0) 0x7fbb64003520 con 0x7fbb780a6c50 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbb780a46a0 con 0x7fbb780a6c50 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbb70002070 con 0x7fbb780a6c50 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): dump: 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2016664063 0 0) 0x7fbb780a46a0 con 0x7fbb780a6c50 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.121:6793/0 conn(0x7fbb780a6160 legacy=0x7fbb781d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.121:6791/0 conn(0x7fbb7809d150 legacy=0x7fbb78139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 monclient: found mon.f 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb781262b0 con 0x7fbb780a6c50 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient: _finish_auth 0 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131338854 0 0) 0x7fbb781247f0 con 0x7fbb7809d150 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb7dca9640 5 monclient: authenticate success, global_id 4249 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb5ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb7dca9640 10 monclient: _renew_subs 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb7dca9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.055+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbb781398a0 con 0x7fbb780a6c50 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 10 monclient: _renew_subs 2024-09-13T13:51:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbb781cd9d0 con 0x7fbb780a6c50 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbb70003da0 con 0x7fbb780a6c50 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb5ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbb70004310 con 0x7fbb780a6c50 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb75ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb5ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb5ffff640 10 monclient: dump: 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.072 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.056+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb58005080 con 0x7fbb780a6c50 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.057+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fbb700044b0 con 0x7fbb780a6c50 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.057+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(16..16 src has 1..16) ==== 3153+0+0 (unknown 39099219 0 0) 0x7fbb7004e370 con 0x7fbb780a6c50 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.057+0000 7fbb777fe640 10 monclient: get_auth_request con 0x7fbb6403fa60 auth_method 0 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.062+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fbb70017a20 con 0x7fbb780a6c50 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.062+0000 7fbb5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:40.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.062+0000 7fbb5ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:40.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.237+0000 7fbb7dca9640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T13:51:40.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.237+0000 7fbb7dca9640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T13:51:40.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.237+0000 7fbb7dca9640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:40.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.237+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 --> v1:172.21.15.153:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fbb58005c40 con 0x7fbb780a6c50 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.450+0000 7fbb5ffff640 1 -- 172.21.15.121:0/1708528230 <== mon.1 v1:172.21.15.153:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) ==== 109+0+0 (unknown 1910661782 0 0) 0x7fbb7001bb90 con 0x7fbb780a6c50 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.450+0000 7fbb5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.450+0000 7fbb5ffff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-13T13:51:40.452 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'rbd' created 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.454+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.153:6812/1206688214 conn(0x7fbb6403fa60 legacy=0x7fbb64041f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.454+0000 7fbb7dca9640 10 monclient: shutdown 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.454+0000 7fbb7dca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.454+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 >> v1:172.21.15.153:6789/0 conn(0x7fbb780a6c50 legacy=0x7fbb781d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.454+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 shutdown_connections 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.454+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 >> 172.21.15.121:0/1708528230 conn(0x7fbb780ab940 msgr2=0x7fbb781c1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.455+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 shutdown_connections 2024-09-13T13:51:40.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.455+0000 7fbb7dca9640 1 -- 172.21.15.121:0/1708528230 wait complete. 2024-09-13T13:51:40.468 DEBUG:teuthology.orchestra.run.smithi121:> rbd --cluster ceph pool init rbd 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.534+0000 7f9d469cef00 10 monclient: get_monmap_and_config 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.534+0000 7f9d469cef00 10 monclient: build_initial_monmap 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.534+0000 7f9d469cef00 10 monclient: monmap: 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:40.536107+0000 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:40.536107+0000 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:40.537 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 1 Processor -- start 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 1 -- start start 2024-09-13T13:51:40.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient: init 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient: _reopen_session rank -1 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient: _add_conns ranks=[2,8,3,0,7,4,5,6,1] 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient(hunting): picked mon.noname-b con 0x5561cbcb4330 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient(hunting): picked mon.noname-e con 0x5561cbc041e0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient(hunting): picked mon.noname-g con 0x5561cbc04630 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5561cba7efe0 con 0x5561cbcb4330 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5561cba87f30 con 0x5561cbc04630 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5561cba6d0c0 con 0x5561cbc041e0 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient(hunting): _renew_subs 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d469cef00 10 monclient(hunting): authenticate will time out at 780.165672s 2024-09-13T13:51:40.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d44cb9640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x5561cbc041e0 0x5561cbcbc3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:48962/0 (socket says 172.21.15.121:48962) 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d454ba640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x5561cbcb4330 0x5561cbcb4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:53818/0 (socket says 172.21.15.121:53818) 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.535+0000 7f9d44cb9640 1 -- 172.21.15.121:0/4274625585 learned_addr learned my addr 172.21.15.121:0/4274625585 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.537+0000 7f9d44cb9640 1 -- 172.21.15.121:0/4274625585 >> v1:172.21.15.121:6793/0 conn(0x5561cbc041e0 legacy=0x5561cbcbc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.537+0000 7f9d454ba640 1 -- 172.21.15.121:0/4274625585 >> v1:172.21.15.121:6790/0 conn(0x5561cbcb4330 legacy=0x5561cbcb4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d4671c640 1 -- 172.21.15.121:0/4274625585 >> v1:172.21.15.153:6790/0 conn(0x5561cbc04630 legacy=0x5561cbcbfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2767379458 0 0) 0x5561cba6d0c0 con 0x5561cbc041e0 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): my global_id is 4247 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d2c001e10 con 0x5561cbc041e0 2024-09-13T13:51:40.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1743071652 0 0) 0x5561cba7efe0 con 0x5561cbcb4330 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): my global_id is 4214 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5561cba6d0c0 con 0x5561cbcb4330 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273186300 0 0) 0x5561cba87f30 con 0x5561cbc04630 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient(hunting): my global_id is 4215 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5561cba7efe0 con 0x5561cbc04630 2024-09-13T13:51:40.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 268580660 0 0) 0x7f9d2c001e10 con 0x5561cbc041e0 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 >> v1:172.21.15.153:6790/0 conn(0x5561cbc04630 legacy=0x5561cbcbfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 >> v1:172.21.15.121:6790/0 conn(0x5561cbcb4330 legacy=0x5561cbcb4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 monclient: found mon.noname-e 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561cbc23e00 con 0x5561cbc041e0 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: _finish_auth 0 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9d30001ae0 con 0x5561cbc041e0 2024-09-13T13:51:40.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: dump: 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.548 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d469cef00 5 monclient: authenticate success, global_id 4247 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d469cef00 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d30003190 con 0x5561cbc041e0 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 1 -- 172.21.15.121:0/4274625585 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9d30003700 con 0x5561cbc041e0 2024-09-13T13:51:40.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d37fff640 10 monclient: dump: 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.550 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d469cef00 10 monclient: get_monmap_and_config success 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.538+0000 7f9d469cef00 4 set_mon_vals no callback set 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 10 monclient: shutdown 2024-09-13T13:51:40.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 -- 172.21.15.121:0/4274625585 >> v1:172.21.15.121:6793/0 conn(0x5561cbc041e0 legacy=0x5561cbcbc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 -- 172.21.15.121:0/4274625585 shutdown_connections 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 -- 172.21.15.121:0/4274625585 >> 172.21.15.121:0/4274625585 conn(0x5561cbc282b0 msgr2=0x5561cbc24150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 -- 172.21.15.121:0/4274625585 shutdown_connections 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 -- 172.21.15.121:0/4274625585 wait complete. 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 10 monclient: build_initial_monmap 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 10 monclient: monmap: 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:40.552 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:40.553 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 Processor -- start 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 1 -- start start 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.539+0000 7f9d469cef00 10 monclient: init 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient: _reopen_session rank -1 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient: _add_conns ranks=[1,8,7,5,2,3,0,4,6] 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient(hunting): picked mon.noname-f con 0x5561cbc041e0 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient(hunting): picked mon.noname-e con 0x5561cbc04630 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient(hunting): picked mon.noname-i con 0x5561cbcb4330 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5561cbc21230 con 0x5561cbc041e0 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5561cbc9f310 con 0x5561cbcb4330 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5561cbc9e030 con 0x5561cbc04630 2024-09-13T13:51:40.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient(hunting): _renew_subs 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d469cef00 10 monclient(hunting): authenticate will time out at 780.170422s 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d44cb9640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x5561cbc04630 0x5561cbc20b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:48972/0 (socket says 172.21.15.121:48972) 2024-09-13T13:51:40.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.540+0000 7f9d44cb9640 1 -- 172.21.15.121:0/703026530 learned_addr learned my addr 172.21.15.121:0/703026530 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:40.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d44cb9640 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.121:6793/0 conn(0x5561cbc04630 legacy=0x5561cbc20b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d454ba640 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.153:6789/0 conn(0x5561cbc041e0 legacy=0x5561cbc20410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506669337 0 0) 0x5561cbc9e030 con 0x5561cbc04630 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): my global_id is 4256 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d24001e10 con 0x5561cbc04630 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.153:6792/0 conn(0x5561cbcb4330 legacy=0x5561cbcce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1390992263 0 0) 0x5561cbc21230 con 0x5561cbc041e0 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 10 monclient(hunting): my global_id is 4258 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x5561cbc9e030 con 0x5561cbc041e0 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2885160069 0 0) 0x7f9d24001e10 con 0x5561cbc04630 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.542+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x5561cbc21230 con 0x5561cbc04630 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2150631645 0 0) 0x5561cbc9f310 con 0x5561cbcb4330 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): my global_id is 4264 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d24001e10 con 0x5561cbcb4330 2024-09-13T13:51:40.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9d30003680 con 0x5561cbc04630 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): dump: 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.562 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2453174803 0 0) 0x5561cbc9e030 con 0x5561cbc041e0 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x5561cbc9f310 con 0x5561cbc041e0 2024-09-13T13:51:40.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9d3c001d10 con 0x5561cbc041e0 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient(hunting): dump: 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.564 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2597746238 0 0) 0x5561cbc21230 con 0x5561cbc04630 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.153:6792/0 conn(0x5561cbcb4330 legacy=0x5561cbcce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.153:6789/0 conn(0x5561cbc041e0 legacy=0x5561cbc20410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:40.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: _finish_hunting 0 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 monclient: found mon.e 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561cbca0390 con 0x5561cbc04630 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: _finish_auth 0 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: _check_auth_tickets 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 5 monclient: authenticate success, global_id 4256 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 10 monclient: _renew_subs 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x5561cbc9ee60 con 0x5561cbc04630 2024-09-13T13:51:40.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 10 monclient: _renew_subs 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x5561cbcb3b30 con 0x5561cbc04630 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d30001d50 con 0x5561cbc04630 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9d30001b00 con 0x5561cbc04630 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d357fa640 10 monclient: dump: 2024-09-13T13:51:40.567 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:40.568 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.543+0000 7f9d377fe640 4 set_mon_vals no callback set 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f9d30010510 con 0x5561cbc04630 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(17..17 src has 1..17) ==== 3495+0+0 (unknown 2939581931 0 0) 0x7f9d3004ceb0 con 0x5561cbc04630 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d469cef00 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d469cef00 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d469cef00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x5561cbccfbd0 con 0x5561cbc04630 2024-09-13T13:51:40.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:40.544+0000 7f9d454ba640 10 monclient: get_auth_request con 0x7f9d2403ebb0 auth_method 0 2024-09-13T13:51:41.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v18) ==== 141+0+0 (unknown 660702925 0 0) 0x7f9d30016470 con 0x5561cbc04630 2024-09-13T13:51:41.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-13T13:51:41.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d357fa640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d469cef00 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f9d30003190 con 0x5561cbc04630 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 10 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (unknown 77724187 0 0) 0x7f9d30015bb0 con 0x5561cbc04630 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d357fa640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d377fe640 10 monclient: _renew_subs 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d377fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d377fe640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=18}) -- 0x7f9d1c002bc0 con 0x5561cbc04630 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 11 ==== osd_map(18..18 src has 1..18) ==== 658+0+0 (unknown 2264769680 0 0) 0x7f9d3001eba0 con 0x5561cbc04630 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.453+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6804/3215471673 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x5561cbcd4270 con 0x5561cbcd0760 2024-09-13T13:51:41.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.454+0000 7f9d4671c640 10 monclient: get_auth_request con 0x5561cbcd0760 auth_method 0 2024-09-13T13:51:41.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.459+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 <== osd.5 v1:172.21.15.153:6804/3215471673 1 ==== osd_op_reply(1 rbd_trash [create] v18'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 785457554 0 0) 0x7f9d38001d60 con 0x5561cbcd0760 2024-09-13T13:51:41.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.462+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6804/3215471673 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) -- 0x5561cbd266a0 con 0x5561cbcd0760 2024-09-13T13:51:41.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.462+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 <== osd.5 v1:172.21.15.153:6804/3215471673 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 1898125418 0 0) 0x7f9d38001d60 con 0x5561cbcd0760 2024-09-13T13:51:41.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.463+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6804/3215471673 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) -- 0x5561cbd257b0 con 0x5561cbcd0760 2024-09-13T13:51:41.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.463+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 <== osd.5 v1:172.21.15.153:6804/3215471673 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 1886642790 0 0) 0x7f9d38001d60 con 0x5561cbcd0760 2024-09-13T13:51:41.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.463+0000 7f9d36ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:41.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:41.463+0000 7f9d36ffd640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f9d20001f00 con 0x5561cbc04630 2024-09-13T13:51:42.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.463+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 12 ==== pool_op_reply(tid 4 (0) Success v19) ==== 55+0+0 (unknown 1417740628 0 0) 0x7f9d3001a580 con 0x5561cbc04630 2024-09-13T13:51:42.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.463+0000 7f9d357fa640 10 monclient: _renew_subs 2024-09-13T13:51:42.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.463+0000 7f9d357fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:42.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.463+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=19}) -- 0x5561cbc21230 con 0x5561cbc04630 2024-09-13T13:51:42.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.463+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 13 ==== osd_map(19..19 src has 1..19) ==== 626+0+0 (unknown 630730185 0 0) 0x7f9d3001edc0 con 0x5561cbc04630 2024-09-13T13:51:42.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.464+0000 7f9d377fe640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6804/3215471673 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f9d1c0058d0 con 0x5561cbcd0760 2024-09-13T13:51:42.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.465+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 <== osd.5 v1:172.21.15.153:6804/3215471673 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v19'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 1570664900 0 0) 0x7f9d38001d60 con 0x5561cbcd0760 2024-09-13T13:51:42.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.465+0000 7f9d36ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:42.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:42.465+0000 7f9d36ffd640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v19) -- 0x7f9d20003370 con 0x5561cbc04630 2024-09-13T13:51:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.467+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 14 ==== pool_op_reply(tid 6 (0) Success v20) ==== 43+0+0 (unknown 2584727254 0 0) 0x7f9d3001f030 con 0x5561cbc04630 2024-09-13T13:51:43.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.467+0000 7f9d357fa640 10 monclient: _renew_subs 2024-09-13T13:51:43.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.467+0000 7f9d357fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:43.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.467+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=20}) -- 0x5561cbc9e030 con 0x5561cbc04630 2024-09-13T13:51:43.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.467+0000 7f9d357fa640 1 -- 172.21.15.121:0/703026530 <== mon.8 v1:172.21.15.121:6793/0 15 ==== osd_map(20..20 src has 1..20) ==== 654+0+0 (unknown 2170786122 0 0) 0x7f9d30014e00 con 0x5561cbc04630 2024-09-13T13:51:43.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.468+0000 7f9d377fe640 1 -- 172.21.15.121:0/703026530 --> v1:172.21.15.153:6804/3215471673 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x7f9d1c005cb0 con 0x5561cbcd0760 2024-09-13T13:51:43.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.470+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 <== osd.5 v1:172.21.15.153:6804/3215471673 5 ==== osd_op_reply(7 rbd_info [write 0~19] v20'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 1721084249 0 0) 0x7f9d38001d60 con 0x5561cbcd0760 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.153:6804/3215471673 conn(0x5561cbcd0760 legacy=0x5561cbcd2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.153:6812/1206688214 conn(0x7f9d2403ebb0 legacy=0x7f9d24041070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 10 monclient: shutdown 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 >> v1:172.21.15.121:6793/0 conn(0x5561cbc04630 legacy=0x5561cbc20b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d4671c640 1 -- 172.21.15.121:0/703026530 reap_dead start 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 shutdown_connections 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 >> 172.21.15.121:0/703026530 conn(0x5561cbc282b0 msgr2=0x5561cbc1df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 shutdown_connections 2024-09-13T13:51:43.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.471+0000 7f9d469cef00 1 -- 172.21.15.121:0/703026530 wait complete. 2024-09-13T13:51:43.476 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T13:51:43.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-13T13:51:43.477 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.580+0000 7f2cf9556640 10 monclient: get_monmap_and_config 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.580+0000 7f2cf9556640 10 monclient: build_initial_monmap 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.580+0000 7f2cf9556640 10 monclient: monmap: 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:43.582131+0000 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:43.582131+0000 2024-09-13T13:51:43.582 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:43.583 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.580+0000 7f2cf9556640 1 Processor -- start 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 1 -- start start 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient: init 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient: _add_conns ranks=[7,3,2,8,5,6,0,4,1] 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient(hunting): picked mon.noname-i con 0x7f2cf40a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient(hunting): picked mon.noname-g con 0x7f2cf409d150 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf40a6c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf409e210 con 0x7f2cf40a6c50 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf40a7ef0 con 0x7f2cf409d150 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf409d970 con 0x7f2cf40a6160 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:43.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf9556640 10 monclient(hunting): authenticate will time out at 783.211546s 2024-09-13T13:51:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf37fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f2cf40a6c50 0x7f2cf4141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59094/0 (socket says 172.21.15.121:59094) 2024-09-13T13:51:43.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.581+0000 7f2cf37fe640 1 -- 172.21.15.121:0/1750392424 learned_addr learned my addr 172.21.15.121:0/1750392424 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:43.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.583+0000 7f2cf37fe640 1 -- 172.21.15.121:0/1750392424 >> v1:172.21.15.121:6790/0 conn(0x7f2cf40a6c50 legacy=0x7f2cf4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.583+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 577776871 0 0) 0x7f2cf409e210 con 0x7f2cf40a6c50 2024-09-13T13:51:43.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient(hunting): my global_id is 4223 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ce0003520 con 0x7f2cf40a6c50 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf27fc640 1 -- 172.21.15.121:0/1750392424 >> v1:172.21.15.153:6790/0 conn(0x7f2cf409d150 legacy=0x7f2cf413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf2ffd640 1 -- 172.21.15.121:0/1750392424 >> v1:172.21.15.153:6792/0 conn(0x7f2cf40a6160 legacy=0x7f2cf40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2488161950 0 0) 0x7f2ce0003520 con 0x7f2cf40a6c50 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 >> v1:172.21.15.153:6792/0 conn(0x7f2cf40a6160 legacy=0x7f2cf40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 >> v1:172.21.15.153:6790/0 conn(0x7f2cf409d150 legacy=0x7f2cf413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 monclient: found mon.noname-b 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf409e630 con 0x7f2cf40a6c50 2024-09-13T13:51:43.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2cec001e60 con 0x7f2cf40a6c50 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: dump: 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.591 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 5 monclient: authenticate success, global_id 4223 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2cec004c20 con 0x7f2cf40a6c50 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:43.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 1 -- 172.21.15.121:0/1750392424 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2cec005150 con 0x7f2cf40a6c50 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 4 set_mon_vals no callback set 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf1ffb640 10 monclient: dump: 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.593 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 10 monclient: shutdown 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 1 -- 172.21.15.121:0/1750392424 >> v1:172.21.15.121:6790/0 conn(0x7f2cf40a6c50 legacy=0x7f2cf4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 1 -- 172.21.15.121:0/1750392424 shutdown_connections 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 1 -- 172.21.15.121:0/1750392424 >> 172.21.15.121:0/1750392424 conn(0x7f2cf40ab940 msgr2=0x7f2cf40abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:43.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.584+0000 7f2cf9556640 1 -- 172.21.15.121:0/1750392424 shutdown_connections 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.585+0000 7f2cf9556640 1 -- 172.21.15.121:0/1750392424 wait complete. 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.585+0000 7f2cf9556640 10 monclient: build_initial_monmap 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.585+0000 7f2cf9556640 10 monclient: monmap: 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:43.595 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 1 Processor -- start 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 1 -- start start 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient: init 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:43.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient: _add_conns ranks=[4,6,8,7,0,5,1,3,2] 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf409d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient(hunting): picked mon.noname-d con 0x7f2cf40a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient(hunting): picked mon.noname-e con 0x7f2cf40a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf4125ae0 con 0x7f2cf409d150 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf41cbc30 con 0x7f2cf40a6160 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf41249c0 con 0x7f2cf40a6c50 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf9556640 10 monclient(hunting): authenticate will time out at 783.216626s 2024-09-13T13:51:43.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf2ffd640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f2cf409d150 0x7f2cf41395b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33666/0 (socket says 172.21.15.121:33666) 2024-09-13T13:51:43.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.586+0000 7f2cf2ffd640 1 -- 172.21.15.121:0/509918385 learned_addr learned my addr 172.21.15.121:0/509918385 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.588+0000 7f2cf2ffd640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.121:6791/0 conn(0x7f2cf409d150 legacy=0x7f2cf41395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.588+0000 7f2cf37fe640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.121:6793/0 conn(0x7f2cf40a6c50 legacy=0x7f2cf41d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.588+0000 7f2cf27fc640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.121:6792/0 conn(0x7f2cf40a6160 legacy=0x7f2cf41d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.588+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263343829 0 0) 0x7f2cf4125ae0 con 0x7f2cf409d150 2024-09-13T13:51:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): my global_id is 4198 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd4003520 con 0x7f2cf409d150 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936900934 0 0) 0x7f2cf41249c0 con 0x7f2cf40a6c50 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): my global_id is 4265 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cf4125ae0 con 0x7f2cf40a6c50 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1579600361 0 0) 0x7f2cf40a7ef0 con 0x7f2cf40a6160 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): my global_id is 4236 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cf41249c0 con 0x7f2cf40a6160 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3622816487 0 0) 0x7f2cf41249c0 con 0x7f2cf40a6160 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cf40a7ef0 con 0x7f2cf40a6160 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2cdc001cd0 con 0x7f2cf40a6160 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.603 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:43.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 82703936 0 0) 0x7f2cf4125ae0 con 0x7f2cf40a6c50 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cf41249c0 con 0x7f2cf40a6c50 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4076248226 0 0) 0x7f2cd4003520 con 0x7f2cf409d150 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cf4125ae0 con 0x7f2cf409d150 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2cec0020b0 con 0x7f2cf40a6c50 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.605 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2ce4003610 con 0x7f2cf409d150 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:43.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.607 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2204283277 0 0) 0x7f2cf40a7ef0 con 0x7f2cf40a6160 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.121:6793/0 conn(0x7f2cf40a6c50 legacy=0x7f2cf41d61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.121:6791/0 conn(0x7f2cf409d150 legacy=0x7f2cf41395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 monclient: found mon.d 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf4139cc0 con 0x7f2cf40a6160 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:43.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 5 monclient: authenticate success, global_id 4236 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 10 monclient: _renew_subs 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf4126d20 con 0x7f2cf40a6160 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 10 monclient: _renew_subs 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2cf4125700 con 0x7f2cf40a6160 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2cdc0039c0 con 0x7f2cf40a6160 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2cdc003eb0 con 0x7f2cf40a6160 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2cf17fa640 4 set_mon_vals no callback set 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:43.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.589+0000 7f2ceb7fe640 10 monclient: dump: 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.610 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2cf9556640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2cf9556640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f2cdc004050 con 0x7f2cf40a6160 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2cf9556640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd8005080 con 0x7f2cf40a6160 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f2cdc04e150 con 0x7f2cf40a6160 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.590+0000 7f2cf2ffd640 10 monclient: get_auth_request con 0x7f2cd4040720 auth_method 0 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.595+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f2cf41cbc30 con 0x7f2cf40a6160 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.595+0000 7f2ceb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:43.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.595+0000 7f2ceb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:43.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.769+0000 7f2cf9556640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T13:51:43.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.769+0000 7f2cf9556640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T13:51:43.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.769+0000 7f2cf9556640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:43.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.769+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f2cd8005370 con 0x7f2cf40a6160 2024-09-13T13:51:43.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.769+0000 7f2ceb7fe640 1 -- 172.21.15.121:0/509918385 <== mon.6 v1:172.21.15.121:6792/0 10 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v4) ==== 86+0+159 (unknown 1197360318 0 2054746940) 0x7f2cdc017730 con 0x7f2cf40a6160 2024-09-13T13:51:43.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.770+0000 7f2ceb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-13T13:51:43.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.770+0000 7f2ceb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:43.772 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.153:6812/1206688214 conn(0x7f2cd4040720 legacy=0x7f2cd4042be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 10 monclient: shutdown 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 >> v1:172.21.15.121:6792/0 conn(0x7f2cf40a6160 legacy=0x7f2cf41d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 shutdown_connections 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 >> 172.21.15.121:0/509918385 conn(0x7f2cf40ab940 msgr2=0x7f2cf409ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 shutdown_connections 2024-09-13T13:51:43.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.772+0000 7f2cf9556640 1 -- 172.21.15.121:0/509918385 wait complete. 2024-09-13T13:51:43.784 INFO:teuthology.orchestra.run.smithi121.stdout:[{"version":4,"timestamp":"2024-09-13T13:51:35.274323+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"9386.070871"}]}] 2024-09-13T13:51:43.784 INFO:tasks.ceph_manager:config epoch is 4 2024-09-13T13:51:43.784 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T13:51:43.784 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T13:51:43.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.880+0000 7f1eddfc2640 10 monclient: get_monmap_and_config 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.880+0000 7f1eddfc2640 10 monclient: build_initial_monmap 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.880+0000 7f1eddfc2640 10 monclient: monmap: 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:43.881942+0000 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:43.881942+0000 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:43.882 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:43.883 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.880+0000 7f1eddfc2640 1 Processor -- start 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 1 -- start start 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient: init 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient: _add_conns ranks=[3,0,5,7,2,6,8,4,1] 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient(hunting): picked mon.noname-g con 0x7f1ed8138c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient(hunting): picked mon.noname-a con 0x7f1ed813bfc0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient(hunting): picked mon.noname-h con 0x7f1ed813fad0 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed80a5e80 con 0x7f1ed813bfc0 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed8139a60 con 0x7f1ed8138c50 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed809df00 con 0x7f1ed813fad0 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1eddfc2640 10 monclient(hunting): authenticate will time out at 783.511635s 2024-09-13T13:51:43.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1ed6ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1ed813bfc0 0x7f1ed813e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45408/0 (socket says 172.21.15.121:45408) 2024-09-13T13:51:43.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.881+0000 7f1ed6ffd640 1 -- 172.21.15.121:0/1810417735 learned_addr learned my addr 172.21.15.121:0/1810417735 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.883+0000 7f1ed6ffd640 1 -- 172.21.15.121:0/1810417735 >> v1:172.21.15.121:6789/0 conn(0x7f1ed813bfc0 legacy=0x7f1ed813e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970550744 0 0) 0x7f1ec0001450 con 0x7f1ed813bfc0 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed77fe640 1 -- 172.21.15.121:0/1810417735 >> v1:172.21.15.153:6790/0 conn(0x7f1ed8138c50 legacy=0x7f1ed80a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): my global_id is 4284 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ec4003520 con 0x7f1ed813bfc0 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed7fff640 1 -- 172.21.15.121:0/1810417735 >> v1:172.21.15.153:6791/0 conn(0x7f1ed813fad0 legacy=0x7f1ed8141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688950525 0 0) 0x7f1ed8139a60 con 0x7f1ed8138c50 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient(hunting): my global_id is 4233 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ec0001450 con 0x7f1ed8138c50 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 33796359 0 0) 0x7f1ec4003520 con 0x7f1ed813bfc0 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 >> v1:172.21.15.153:6791/0 conn(0x7f1ed813fad0 legacy=0x7f1ed8141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 >> v1:172.21.15.153:6790/0 conn(0x7f1ed8138c50 legacy=0x7f1ed80a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 monclient: found mon.noname-a 2024-09-13T13:51:43.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed809d940 con 0x7f1ed813bfc0 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 5 monclient: authenticate success, global_id 4284 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec0001cb0 con 0x7f1ed813bfc0 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: dump: 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.891 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ec0004950 con 0x7f1ed813bfc0 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 1 -- 172.21.15.121:0/1810417735 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec0004e80 con 0x7f1ed813bfc0 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 4 set_mon_vals no callback set 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1ed67fc640 10 monclient: dump: 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.893 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 10 monclient: shutdown 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1810417735 >> v1:172.21.15.121:6789/0 conn(0x7f1ed813bfc0 legacy=0x7f1ed813e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1810417735 shutdown_connections 2024-09-13T13:51:43.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1810417735 >> 172.21.15.121:0/1810417735 conn(0x7f1ed8069360 msgr2=0x7f1ed80ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1810417735 shutdown_connections 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.884+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1810417735 wait complete. 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.885+0000 7f1eddfc2640 10 monclient: build_initial_monmap 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.885+0000 7f1eddfc2640 10 monclient: monmap: 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:43.895 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.885+0000 7f1eddfc2640 1 Processor -- start 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.885+0000 7f1eddfc2640 1 -- start start 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.885+0000 7f1eddfc2640 10 monclient: init 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient: _add_conns ranks=[4,1,0,6,3,5,8,7,2] 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient(hunting): picked mon.noname-c con 0x7f1ed8138c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:43.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient(hunting): picked mon.noname-f con 0x7f1ed813bfc0 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient(hunting): picked mon.noname-a con 0x7f1ed813fad0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed80a0d60 con 0x7f1ed813fad0 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed80a1040 con 0x7f1ed813bfc0 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed8132d70 con 0x7f1ed8138c50 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1eddfc2640 10 monclient(hunting): authenticate will time out at 783.516434s 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1ed77fe640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f1ed8138c50 0x7f1ed8131f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33668/0 (socket says 172.21.15.121:33668) 2024-09-13T13:51:43.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.886+0000 7f1ed77fe640 1 -- 172.21.15.121:0/1349043763 learned_addr learned my addr 172.21.15.121:0/1349043763 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:43.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ed77fe640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.121:6791/0 conn(0x7f1ed8138c50 legacy=0x7f1ed8131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ed7fff640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.121:6789/0 conn(0x7f1ed813fad0 legacy=0x7f1ed81d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3420597273 0 0) 0x7f1ed8132d70 con 0x7f1ed8138c50 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): my global_id is 4207 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb8003520 con 0x7f1ed8138c50 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1720356158 0 0) 0x7f1ed80a0d60 con 0x7f1ed813fad0 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 10 monclient(hunting): my global_id is 4293 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed8132d70 con 0x7f1ed813fad0 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.888+0000 7f1ed6ffd640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.153:6789/0 conn(0x7f1ed813bfc0 legacy=0x7f1ed8132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4033219565 0 0) 0x7f1eb8003520 con 0x7f1ed8138c50 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ed80a0d60 con 0x7f1ed8138c50 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec8003610 con 0x7f1ed8138c50 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): dump: 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.903 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 14333095 0 0) 0x7f1ed8132d70 con 0x7f1ed813fad0 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eb8003520 con 0x7f1ed813fad0 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ed0003990 con 0x7f1ed813fad0 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): dump: 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.905 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101990276 0 0) 0x7f1ed80a1040 con 0x7f1ed813bfc0 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:43.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient(hunting): my global_id is 4267 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed8132d70 con 0x7f1ed813bfc0 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1887666489 0 0) 0x7f1ed80a0d60 con 0x7f1ed8138c50 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.153:6789/0 conn(0x7f1ed813bfc0 legacy=0x7f1ed8132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.121:6789/0 conn(0x7f1ed813fad0 legacy=0x7f1ed81d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 monclient: found mon.c 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed80a56e0 con 0x7f1ed8138c50 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: _finish_auth 0 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 5 monclient: authenticate success, global_id 4207 2024-09-13T13:51:43.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 10 monclient: _renew_subs 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed80a5b80 con 0x7f1ed8138c50 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 10 monclient: _renew_subs 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ed80a07a0 con 0x7f1ed8138c50 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ec8003b90 con 0x7f1ed8138c50 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec80040c0 con 0x7f1ed8138c50 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ed5ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.889+0000 7f1ecffff640 10 monclient: dump: 2024-09-13T13:51:43.908 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:43.909 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1eddfc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1eddfc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1eddfc2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ebc005080 con 0x7f1ed8138c50 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1ec8004260 con 0x7f1ed8138c50 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f1ec804e1b0 con 0x7f1ed8138c50 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.890+0000 7f1ed6ffd640 10 monclient: get_auth_request con 0x7f1eb8040090 auth_method 0 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.895+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f1ec8017870 con 0x7f1ed8138c50 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.895+0000 7f1ecffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:43.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:43.895+0000 7f1ecffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:44.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.115+0000 7f1eddfc2640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-13T13:51:44.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.115+0000 7f1eddfc2640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-13T13:51:44.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.115+0000 7f1eddfc2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:44.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.115+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f1ebc005370 con 0x7f1ed8138c50 2024-09-13T13:51:44.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.117+0000 7f1ecffff640 1 -- 172.21.15.121:0/1349043763 <== mon.4 v1:172.21.15.121:6791/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+96984 (unknown 481731318 0 354480859) 0x7f1ec8017170 con 0x7f1ed8138c50 2024-09-13T13:51:44.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.117+0000 7f1ecffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-13T13:51:44.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.117+0000 7f1ecffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:44.120 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:44.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1eb8040090 legacy=0x7f1eb8042550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 10 monclient: shutdown 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 >> v1:172.21.15.121:6791/0 conn(0x7f1ed8138c50 legacy=0x7f1ed8131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 shutdown_connections 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 >> 172.21.15.121:0/1349043763 conn(0x7f1ed8069360 msgr2=0x7f1ed8139d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 shutdown_connections 2024-09-13T13:51:44.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.119+0000 7f1eddfc2640 1 -- 172.21.15.121:0/1349043763 wait complete. 2024-09-13T13:51:44.131 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":5,"flags":0,"active_gid":4121,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6812","nonce":1206688214}]},"active_addr":"172.21.15.153:6812/1206688214","active_change":"2024-09-13T13:51:27.208415+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3163727654}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":4201384809}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":138702282}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3743366213}]}]} 2024-09-13T13:51:44.134 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T13:51:44.134 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T13:51:44.134 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.229+0000 7f3148b4b640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.229+0000 7f3148b4b640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.229+0000 7f3148b4b640 10 monclient: monmap: 2024-09-13T13:51:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.231374+0000 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.231374+0000 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.233 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 1 Processor -- start 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 1 -- start start 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 10 monclient: init 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 10 monclient: _add_conns ranks=[2,4,5,7,1,3,8,0,6] 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f314409c950 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f3144135b50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 10 monclient(hunting): picked mon.noname-h con 0x7f314413f660 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.230+0000 7f3148b4b640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3144135f50 con 0x7f314409c950 2024-09-13T13:51:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.231+0000 7f3148b4b640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314409d860 con 0x7f3144135b50 2024-09-13T13:51:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.231+0000 7f3148b4b640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314409db40 con 0x7f314413f660 2024-09-13T13:51:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.231+0000 7f3148b4b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.231+0000 7f3148b4b640 10 monclient(hunting): authenticate will time out at 783.861037s 2024-09-13T13:51:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.231+0000 7f3141d9b640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f3144135b50 0x7f314413df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33674/0 (socket says 172.21.15.121:33674) 2024-09-13T13:51:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.231+0000 7f3141d9b640 1 -- 172.21.15.121:0/1938165519 learned_addr learned my addr 172.21.15.121:0/1938165519 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314259c640 1 -- 172.21.15.121:0/1938165519 >> v1:172.21.15.121:6790/0 conn(0x7f314409c950 legacy=0x7f31440a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3141d9b640 1 -- 172.21.15.121:0/1938165519 >> v1:172.21.15.121:6791/0 conn(0x7f3144135b50 legacy=0x7f314413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020081404 0 0) 0x7f3144135f50 con 0x7f314409c950 2024-09-13T13:51:44.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): my global_id is 4232 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f312c003520 con 0x7f314409c950 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360410751 0 0) 0x7f314409d860 con 0x7f3144135b50 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient(hunting): my global_id is 4216 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3144135f50 con 0x7f3144135b50 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3142d9d640 1 -- 172.21.15.121:0/1938165519 >> v1:172.21.15.153:6791/0 conn(0x7f314413f660 legacy=0x7f3144141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1880607070 0 0) 0x7f312c003520 con 0x7f314409c950 2024-09-13T13:51:44.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 >> v1:172.21.15.153:6791/0 conn(0x7f314413f660 legacy=0x7f3144141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 >> v1:172.21.15.121:6791/0 conn(0x7f3144135b50 legacy=0x7f314413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 monclient: found mon.noname-b 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314409de20 con 0x7f314409c950 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f3134001b30 con 0x7f314409c950 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3148b4b640 5 monclient: authenticate success, global_id 4232 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3148b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: dump: 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.242 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3148b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3134003610 con 0x7f314409c950 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3148b4b640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 1 -- 172.21.15.121:0/1938165519 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f3134003b80 con 0x7f314409c950 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f3148b4b640 4 set_mon_vals no callback set 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.233+0000 7f314159a640 10 monclient: dump: 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.243 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 10 monclient: shutdown 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 1 -- 172.21.15.121:0/1938165519 >> v1:172.21.15.121:6790/0 conn(0x7f314409c950 legacy=0x7f31440a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 1 -- 172.21.15.121:0/1938165519 shutdown_connections 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 1 -- 172.21.15.121:0/1938165519 >> 172.21.15.121:0/1938165519 conn(0x7f31440ab760 msgr2=0x7f31440abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 1 -- 172.21.15.121:0/1938165519 shutdown_connections 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.234+0000 7f3148b4b640 1 -- 172.21.15.121:0/1938165519 wait complete. 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient: monmap: 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.245 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 1 Processor -- start 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 1 -- start start 2024-09-13T13:51:44.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient: init 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient: _add_conns ranks=[5,7,1,0,4,3,2,6,8] 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient(hunting): picked mon.noname-h con 0x7f314409c950 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient(hunting): picked mon.noname-i con 0x7f3144135b50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient(hunting): picked mon.noname-f con 0x7f314413f660 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314413c280 con 0x7f314413f660 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3144142a70 con 0x7f314409c950 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314413dd40 con 0x7f3144135b50 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.235+0000 7f3148b4b640 10 monclient(hunting): authenticate will time out at 783.865879s 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.236+0000 7f314259c640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f314409c950 0x7f31441ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:45022/0 (socket says 172.21.15.121:45022) 2024-09-13T13:51:44.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.236+0000 7f314259c640 1 -- 172.21.15.121:0/4070008889 learned_addr learned my addr 172.21.15.121:0/4070008889 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f314259c640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6791/0 conn(0x7f314409c950 legacy=0x7f31441ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3142d9d640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6789/0 conn(0x7f314413f660 legacy=0x7f31441d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3141d9b640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6792/0 conn(0x7f3144135b50 legacy=0x7f31441d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3187296344 0 0) 0x7f3144142a70 con 0x7f314409c950 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): my global_id is 4226 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3124003520 con 0x7f314409c950 2024-09-13T13:51:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3686599090 0 0) 0x7f314413c280 con 0x7f314413f660 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): my global_id is 4276 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3144142a70 con 0x7f314413f660 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446263170 0 0) 0x7f314413dd40 con 0x7f3144135b50 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): my global_id is 4282 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f314413c280 con 0x7f3144135b50 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1935605031 0 0) 0x7f3124003520 con 0x7f314409c950 2024-09-13T13:51:44.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f314413dd40 con 0x7f314409c950 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f3134003b00 con 0x7f314409c950 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.254 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3769577286 0 0) 0x7f314413c280 con 0x7f3144135b50 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3124003520 con 0x7f3144135b50 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2235820651 0 0) 0x7f3144142a70 con 0x7f314413f660 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f314413c280 con 0x7f314413f660 2024-09-13T13:51:44.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f31380047e0 con 0x7f3144135b50 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.256 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f313c001ff0 con 0x7f314413f660 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.257 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2548705240 0 0) 0x7f314413dd40 con 0x7f314409c950 2024-09-13T13:51:44.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.238+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6792/0 conn(0x7f3144135b50 legacy=0x7f31441d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6789/0 conn(0x7f314413f660 legacy=0x7f31441d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 1 monclient: found mon.h 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314413cde0 con 0x7f314409c950 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 5 monclient: authenticate success, global_id 4226 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 monclient: _renew_subs 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f314413ea00 con 0x7f314409c950 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 monclient: _renew_subs 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f314413b980 con 0x7f314409c950 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3134003610 con 0x7f314409c950 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f3134001ad0 con 0x7f314409c950 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3140d99640 4 set_mon_vals no callback set 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3132ffd640 10 monclient: dump: 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.260 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:44.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.239+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3128005080 con 0x7f314409c950 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.240+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f3134011ca0 con 0x7f314409c950 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.240+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f313404e310 con 0x7f314409c950 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.240+0000 7f3141d9b640 10 monclient: get_auth_request con 0x7f3124040580 auth_method 0 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.245+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f31340178f0 con 0x7f314409c950 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.245+0000 7f3132ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:44.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.245+0000 7f3132ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:44.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.427+0000 7f3148b4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:44.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.427+0000 7f3148b4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:44.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.427+0000 7f3148b4b640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.427+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3128005370 con 0x7f314409c950 2024-09-13T13:51:44.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.428+0000 7f3132ffd640 1 -- 172.21.15.121:0/4070008889 <== mon.5 v1:172.21.15.153:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10977 (unknown 3413328019 0 758677432) 0x7f31340171f0 con 0x7f314409c950 2024-09-13T13:51:44.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.428+0000 7f3132ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:44.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.428+0000 7f3132ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:44.430 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:44.430 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":20,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:43.458293+0000","last_up_change":"2024-09-13T13:51:36.267919+0000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:51:35.278046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:51:40.243263+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":3060725185}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":3060725185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":3060725185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":3060725185}]},"public_addr":"172.21.15.121:6808/3060725185","cluster_addr":"172.21.15.121:6809/3060725185","heartbeat_back_addr":"172.21.15.121:6811/3060725185","heartbeat_front_addr":"172.21.15.121:6810/3060725185","state":["exists","up"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":1651307734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1651307734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1651307734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1651307734}]},"public_addr":"172.21.15.121:6800/1651307734","cluster_addr":"172.21.15.121:6801/1651307734","heartbeat_back_addr":"172.21.15.121:6803/1651307734","heartbeat_front_addr":"172.21.15.121:6802/1651307734","state":["exists","up"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2141781943}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2141781943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2141781943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2141781943}]},"public_addr":"172.21.15.121:6804/2141781943","cluster_addr":"172.21.15.121:6805/2141781943","heartbeat_back_addr":"172.21.15.121:6807/2141781943","heartbeat_front_addr":"172.21.15.121:6806/2141781943","state":["exists","up"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6808","nonce":934815702}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6809","nonce":934815702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6811","nonce":934815702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6810","nonce":934815702}]},"public_addr":"172.21.15.153:6808/934815702","cluster_addr":"172.21.15.153:6809/934815702","heartbeat_back_addr":"172.21.15.153:6811/934815702","heartbeat_front_addr":"172.21.15.153:6810/934815702","state":["exists","up"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6800","nonce":1083052486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6801","nonce":1083052486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6803","nonce":1083052486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6802","nonce":1083052486}]},"public_addr":"172.21.15.153:6800/1083052486","cluster_addr":"172.21.15.153:6801/1083052486","heartbeat_back_addr":"172.21.15.153:6803/1083052486","heartbeat_front_addr":"172.21.15.153:6802/1083052486","state":["exists","up"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6804","nonce":3215471673}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6805","nonce":3215471673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6807","nonce":3215471673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6806","nonce":3215471673}]},"public_addr":"172.21.15.153:6804/3215471673","cluster_addr":"172.21.15.153:6805/3215471673","heartbeat_back_addr":"172.21.15.153:6807/3215471673","heartbeat_front_addr":"172.21.15.153:6806/3215471673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.867807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.618229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.154306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.400091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:30.983001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.211095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:44.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6812/1206688214 conn(0x7f3124040580 legacy=0x7f3124042a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 10 monclient: shutdown 2024-09-13T13:51:44.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 >> v1:172.21.15.153:6791/0 conn(0x7f314409c950 legacy=0x7f31441ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 shutdown_connections 2024-09-13T13:51:44.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 >> 172.21.15.121:0/4070008889 conn(0x7f31440ab760 msgr2=0x7f3144133f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 shutdown_connections 2024-09-13T13:51:44.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.430+0000 7f3148b4b640 1 -- 172.21.15.121:0/4070008889 wait complete. 2024-09-13T13:51:44.443 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T13:51:44.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:44.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.538+0000 7feae10ef640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.538+0000 7feae10ef640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.538+0000 7feae10ef640 10 monclient: monmap: 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.540378+0000 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.540378+0000 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.541 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.542 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.542 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 1 Processor -- start 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 1 -- start start 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient: init 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient: _add_conns ranks=[4,6,2,0,5,3,7,8,1] 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient(hunting): picked mon.noname-c con 0x7feadc09c950 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient(hunting): picked mon.noname-d con 0x7feadc135b50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient(hunting): picked mon.noname-b con 0x7feadc13f660 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:44.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc135f50 con 0x7feadc13f660 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc09d860 con 0x7feadc09c950 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc09db40 con 0x7feadc135b50 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feae10ef640 10 monclient(hunting): authenticate will time out at 784.169748s 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feadad9d640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7feadc09c950 0x7feadc0a7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33684/0 (socket says 172.21.15.121:33684) 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feadad9d640 1 -- 172.21.15.121:0/3266569908 learned_addr learned my addr 172.21.15.121:0/3266569908 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.539+0000 7feada59c640 1 --1- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6792/0 conn(0x7feadc135b50 0x7feadc13df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52234/0 (socket says 172.21.15.121:52234) 2024-09-13T13:51:44.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.541+0000 7feadb59e640 1 -- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6790/0 conn(0x7feadc13f660 legacy=0x7feadc141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feadad9d640 1 -- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6791/0 conn(0x7feadc09c950 legacy=0x7feadc0a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feada59c640 1 -- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6792/0 conn(0x7feadc135b50 legacy=0x7feadc13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884980735 0 0) 0x7feadc135f50 con 0x7feadc13f660 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): my global_id is 4241 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac4003520 con 0x7feadc13f660 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237146527 0 0) 0x7feadc09d860 con 0x7feadc09c950 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): my global_id is 4225 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feadc135f50 con 0x7feadc09c950 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136576316 0 0) 0x7feadc09db40 con 0x7feadc135b50 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient(hunting): my global_id is 4245 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feadc09d860 con 0x7feadc135b50 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 523163141 0 0) 0x7feac4003520 con 0x7feadc13f660 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6792/0 conn(0x7feadc135b50 legacy=0x7feadc13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6791/0 conn(0x7feadc09c950 legacy=0x7feadc0a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 monclient: found mon.noname-b 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc09de20 con 0x7feadc13f660 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fead4001eb0 con 0x7feadc13f660 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feae10ef640 5 monclient: authenticate success, global_id 4241 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feae10ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:44.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: dump: 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.551 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feae10ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fead4003990 con 0x7feadc13f660 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 1 -- 172.21.15.121:0/3266569908 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fead4003f00 con 0x7feadc13f660 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7fead9d9b640 10 monclient: dump: 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.552 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feae10ef640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.542+0000 7feae10ef640 4 set_mon_vals no callback set 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 10 monclient: shutdown 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 1 -- 172.21.15.121:0/3266569908 >> v1:172.21.15.121:6790/0 conn(0x7feadc13f660 legacy=0x7feadc141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 1 -- 172.21.15.121:0/3266569908 shutdown_connections 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 1 -- 172.21.15.121:0/3266569908 >> 172.21.15.121:0/3266569908 conn(0x7feadc0ab760 msgr2=0x7feadc0abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 1 -- 172.21.15.121:0/3266569908 shutdown_connections 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 1 -- 172.21.15.121:0/3266569908 wait complete. 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.543+0000 7feae10ef640 10 monclient: monmap: 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.554 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 1 Processor -- start 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 1 -- start start 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient: init 2024-09-13T13:51:44.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient: _add_conns ranks=[5,3,1,8,7,2,6,0,4] 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient(hunting): picked mon.noname-h con 0x7feadc09c950 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient(hunting): picked mon.noname-g con 0x7feadc135b50 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient(hunting): picked mon.noname-f con 0x7feadc13f660 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc1350e0 con 0x7feadc13f660 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc142e20 con 0x7feadc135b50 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feadc13b430 con 0x7feadc09c950 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.544+0000 7feae10ef640 10 monclient(hunting): authenticate will time out at 784.174944s 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.545+0000 7feada59c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7feadc135b50 0x7feadc13ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:47900/0 (socket says 172.21.15.121:47900) 2024-09-13T13:51:44.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.545+0000 7feada59c640 1 -- 172.21.15.121:0/2663185628 learned_addr learned my addr 172.21.15.121:0/2663185628 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feada59c640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6790/0 conn(0x7feadc135b50 legacy=0x7feadc13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feadb59e640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6789/0 conn(0x7feadc13f660 legacy=0x7feadc1d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feadad9d640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6791/0 conn(0x7feadc09c950 legacy=0x7feadc13a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 35844787 0 0) 0x7feadc09de20 con 0x7feadc13f660 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): my global_id is 4285 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac8003520 con 0x7feadc13f660 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032192632 0 0) 0x7fead0001500 con 0x7feadc135b50 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): my global_id is 4242 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feadc09de20 con 0x7feadc135b50 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3418669216 0 0) 0x7feadc13b430 con 0x7feadc09c950 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): my global_id is 4235 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fead0001500 con 0x7feadc09c950 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 288542438 0 0) 0x7fead0001500 con 0x7feadc09c950 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feadc13b430 con 0x7feadc09c950 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 435268481 0 0) 0x7feac8003520 con 0x7feadc13f660 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fead0001500 con 0x7feadc13f660 2024-09-13T13:51:44.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7feacc0047e0 con 0x7feadc09c950 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): dump: 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.562 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2673415456 0 0) 0x7feadc09de20 con 0x7feadc135b50 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feac8003520 con 0x7feadc135b50 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fead4003e80 con 0x7feadc13f660 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): dump: 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.563 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fead0001320 con 0x7feadc135b50 2024-09-13T13:51:44.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.547+0000 7feac37fe640 10 monclient(hunting): dump: 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.565 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1853481982 0 0) 0x7feadc13b430 con 0x7feadc09c950 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6790/0 conn(0x7feadc135b50 legacy=0x7feadc13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6789/0 conn(0x7feadc13f660 legacy=0x7feadc1d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 monclient: found mon.h 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc13ecf0 con 0x7feadc09c950 2024-09-13T13:51:44.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 5 monclient: authenticate success, global_id 4235 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 10 monclient: _renew_subs 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feadc13e050 con 0x7feadc09c950 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 10 monclient: _renew_subs 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7feadc0a53b0 con 0x7feadc09c950 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feacc004c30 con 0x7feadc09c950 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7feacc005120 con 0x7feadc09c950 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7fead959a640 4 set_mon_vals no callback set 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.548+0000 7feac37fe640 10 monclient: dump: 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.568 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feae10ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feae10ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feae10ef640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feabc005080 con 0x7feadc09c950 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7feacc0117a0 con 0x7feadc09c950 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7feacc04e070 con 0x7feadc09c950 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.549+0000 7feada59c640 10 monclient: get_auth_request con 0x7feac8040650 auth_method 0 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.554+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7feacc017650 con 0x7feadc09c950 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.554+0000 7feac37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:44.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.554+0000 7feac37fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:44.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.729+0000 7feae10ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:44.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.729+0000 7feae10ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:44.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.729+0000 7feae10ef640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:44.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.729+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feabc005370 con 0x7feadc09c950 2024-09-13T13:51:44.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.730+0000 7feac37fe640 1 -- 172.21.15.121:0/2663185628 <== mon.5 v1:172.21.15.153:6791/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10977 (unknown 3413328019 0 758677432) 0x7feacc016f50 con 0x7feadc09c950 2024-09-13T13:51:44.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.730+0000 7feac37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:44.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.730+0000 7feac37fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:44.732 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:44.732 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":20,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:43.458293+0000","last_up_change":"2024-09-13T13:51:36.267919+0000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:51:35.278046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:51:40.243263+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":3060725185}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":3060725185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":3060725185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":3060725185}]},"public_addr":"172.21.15.121:6808/3060725185","cluster_addr":"172.21.15.121:6809/3060725185","heartbeat_back_addr":"172.21.15.121:6811/3060725185","heartbeat_front_addr":"172.21.15.121:6810/3060725185","state":["exists","up"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":1651307734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1651307734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1651307734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1651307734}]},"public_addr":"172.21.15.121:6800/1651307734","cluster_addr":"172.21.15.121:6801/1651307734","heartbeat_back_addr":"172.21.15.121:6803/1651307734","heartbeat_front_addr":"172.21.15.121:6802/1651307734","state":["exists","up"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2141781943}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2141781943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2141781943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2141781943}]},"public_addr":"172.21.15.121:6804/2141781943","cluster_addr":"172.21.15.121:6805/2141781943","heartbeat_back_addr":"172.21.15.121:6807/2141781943","heartbeat_front_addr":"172.21.15.121:6806/2141781943","state":["exists","up"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6808","nonce":934815702}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6809","nonce":934815702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6811","nonce":934815702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6810","nonce":934815702}]},"public_addr":"172.21.15.153:6808/934815702","cluster_addr":"172.21.15.153:6809/934815702","heartbeat_back_addr":"172.21.15.153:6811/934815702","heartbeat_front_addr":"172.21.15.153:6810/934815702","state":["exists","up"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6800","nonce":1083052486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6801","nonce":1083052486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6803","nonce":1083052486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6802","nonce":1083052486}]},"public_addr":"172.21.15.153:6800/1083052486","cluster_addr":"172.21.15.153:6801/1083052486","heartbeat_back_addr":"172.21.15.153:6803/1083052486","heartbeat_front_addr":"172.21.15.153:6802/1083052486","state":["exists","up"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6804","nonce":3215471673}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6805","nonce":3215471673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6807","nonce":3215471673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6806","nonce":3215471673}]},"public_addr":"172.21.15.153:6804/3215471673","cluster_addr":"172.21.15.153:6805/3215471673","heartbeat_back_addr":"172.21.15.153:6807/3215471673","heartbeat_front_addr":"172.21.15.153:6806/3215471673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.867807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.618229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.154306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.400091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:30.983001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.211095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.732+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6812/1206688214 conn(0x7feac8040650 legacy=0x7feac8042b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.732+0000 7feae10ef640 10 monclient: shutdown 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.732+0000 7feae10ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.732+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 >> v1:172.21.15.153:6791/0 conn(0x7feadc09c950 legacy=0x7feadc13a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.732+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 shutdown_connections 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.732+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 >> 172.21.15.121:0/2663185628 conn(0x7feadc0ab760 msgr2=0x7feadc0a6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.733+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 shutdown_connections 2024-09-13T13:51:44.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.733+0000 7feae10ef640 1 -- 172.21.15.121:0/2663185628 wait complete. 2024-09-13T13:51:44.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T13:51:44.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T13:51:44.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T13:51:44.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T13:51:44.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T13:51:44.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T13:51:44.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7f6a0e59b640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7fd1bd6bf640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7f6a0e59b640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7f6a0e59b640 10 monclient: monmap: 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.858848+0000 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.858848+0000 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.860 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.861 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.861 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.861 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.861 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.861 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.861 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7fd1bd6bf640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7fd1bd6bf640 10 monclient: monmap: 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.859002+0000 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.859002+0000 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.862 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7fd1bd6bf640 1 Processor -- start 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7fd1bd6bf640 1 -- start start 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.857+0000 7fd1bd6bf640 10 monclient: init 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 1 Processor -- start 2024-09-13T13:51:44.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 1 -- start start 2024-09-13T13:51:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient: init 2024-09-13T13:51:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient: _add_conns ranks=[0,6,5,4,1,8,3,7,2] 2024-09-13T13:51:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient(hunting): picked mon.noname-a con 0x7fd1b80a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient(hunting): picked mon.noname-d con 0x7fd1b809d150 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:44.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1b67fc640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fd1b809d150 0x7fd1b813e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52238/0 (socket says 172.21.15.121:52238) 2024-09-13T13:51:44.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient: _add_conns ranks=[5,0,7,1,2,6,3,8,4] 2024-09-13T13:51:44.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1b67fc640 1 -- 172.21.15.121:0/4146766267 learned_addr learned my addr 172.21.15.121:0/4146766267 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient(hunting): picked mon.noname-h con 0x7f6a080a6160 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0809d150 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient(hunting): picked mon.noname-i con 0x7f6a080a6c50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0809e210 con 0x7f6a0809d150 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a080a7ef0 con 0x7f6a080a6160 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0809d970 con 0x7f6a080a6c50 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7f6a0e59b640 10 monclient(hunting): authenticate will time out at 784.488989s 2024-09-13T13:51:44.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient(hunting): picked mon.noname-h con 0x7fd1b80a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b809e210 con 0x7fd1b80a6160 2024-09-13T13:51:44.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.859+0000 7f6a07fff640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f6a080a6160 0x7f6a080a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:45048/0 (socket says 172.21.15.121:45048) 2024-09-13T13:51:44.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.859+0000 7f6a07fff640 1 -- 172.21.15.121:0/706925840 learned_addr learned my addr 172.21.15.121:0/706925840 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b80a7ef0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b809d970 con 0x7fd1b809d150 2024-09-13T13:51:44.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.858+0000 7fd1bd6bf640 10 monclient(hunting): authenticate will time out at 784.488986s 2024-09-13T13:51:44.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.860+0000 7fd1b67fc640 1 -- 172.21.15.121:0/4146766267 >> v1:172.21.15.121:6792/0 conn(0x7fd1b809d150 legacy=0x7fd1b813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f69fffff640 1 -- 172.21.15.121:0/706925840 >> v1:172.21.15.121:6789/0 conn(0x7f6a0809d150 legacy=0x7f6a0813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b6ffd640 1 -- 172.21.15.121:0/4146766267 >> v1:172.21.15.121:6789/0 conn(0x7fd1b80a6160 legacy=0x7fd1b80a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a07fff640 1 -- 172.21.15.121:0/706925840 >> v1:172.21.15.153:6791/0 conn(0x7f6a080a6160 legacy=0x7f6a080a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a0cb38640 1 -- 172.21.15.121:0/706925840 >> v1:172.21.15.153:6792/0 conn(0x7f6a080a6c50 legacy=0x7f6a08141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618497055 0 0) 0x7f6a080a7ef0 con 0x7f6a080a6160 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): my global_id is 4253 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f4003520 con 0x7f6a080a6160 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885179851 0 0) 0x7f6a0809d970 con 0x7f6a080a6c50 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): my global_id is 4291 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a080a7ef0 con 0x7f6a080a6c50 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399293312 0 0) 0x7f6a0809e210 con 0x7f6a0809d150 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 10 monclient(hunting): my global_id is 4311 2024-09-13T13:51:44.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0809d970 con 0x7f6a0809d150 2024-09-13T13:51:44.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4250929351 0 0) 0x7fd1b809d970 con 0x7fd1b809d150 2024-09-13T13:51:44.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): my global_id is 4254 2024-09-13T13:51:44.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3134022680 0 0) 0x7f6a080a7ef0 con 0x7f6a080a6c50 2024-09-13T13:51:44.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 >> v1:172.21.15.153:6791/0 conn(0x7f6a080a6160 legacy=0x7f6a080a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 >> v1:172.21.15.121:6789/0 conn(0x7f6a0809d150 legacy=0x7f6a0813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 1 monclient: found mon.noname-i 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.153:6792/0 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 --> v1:172.21.15.153:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0809e630 con 0x7f6a080a6c50 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6a00001e90 con 0x7f6a080a6c50 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a077fe640 10 monclient: dump: 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.873 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a0e59b640 5 monclient: authenticate success, global_id 4291 2024-09-13T13:51:44.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7f6a0e59b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b77fe640 1 -- 172.21.15.121:0/4146766267 >> v1:172.21.15.153:6791/0 conn(0x7fd1b80a6c50 legacy=0x7fd1b8141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a0003520 con 0x7fd1b809d150 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 607697327 0 0) 0x7fd1b809e210 con 0x7fd1b80a6160 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient(hunting): my global_id is 4302 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b809d970 con 0x7fd1b80a6160 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2186736843 0 0) 0x7fd1a0003520 con 0x7fd1b809d150 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 >> v1:172.21.15.153:6791/0 conn(0x7fd1b80a6c50 legacy=0x7fd1b8141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 >> v1:172.21.15.121:6789/0 conn(0x7fd1b80a6160 legacy=0x7fd1b80a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 monclient: found mon.noname-d 2024-09-13T13:51:44.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b809e630 con 0x7fd1b809d150 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd1ac001ae0 con 0x7fd1b809d150 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1b5ffb640 10 monclient: dump: 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.879 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1bd6bf640 5 monclient: authenticate success, global_id 4254 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.861+0000 7fd1bd6bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1ac001d30 con 0x7fd1b809d150 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1b5ffb640 1 -- 172.21.15.121:0/4146766267 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd1ac004c60 con 0x7fd1b809d150 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1b5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1b5ffb640 10 monclient: dump: 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.881 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.7 v1:172.21.15.153:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a00003990 con 0x7f6a080a6c50 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a077fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a077fe640 1 -- 172.21.15.121:0/706925840 <== mon.7 v1:172.21.15.153:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6a00003f00 con 0x7f6a080a6c50 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a077fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a077fe640 10 monclient: dump: 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.883 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a0e59b640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7f6a0e59b640 4 set_mon_vals no callback set 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1bd6bf640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.862+0000 7fd1bd6bf640 4 set_mon_vals no callback set 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 10 monclient: shutdown 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 >> v1:172.21.15.121:6792/0 conn(0x7fd1b809d150 legacy=0x7fd1b813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 shutdown_connections 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 >> 172.21.15.121:0/4146766267 conn(0x7fd1b80ab940 msgr2=0x7fd1b80abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 shutdown_connections 2024-09-13T13:51:44.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.863+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/4146766267 wait complete. 2024-09-13T13:51:44.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 10 monclient: shutdown 2024-09-13T13:51:44.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 1 -- 172.21.15.121:0/706925840 >> v1:172.21.15.153:6792/0 conn(0x7f6a080a6c50 legacy=0x7f6a08141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 1 -- 172.21.15.121:0/706925840 shutdown_connections 2024-09-13T13:51:44.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 1 -- 172.21.15.121:0/706925840 >> 172.21.15.121:0/706925840 conn(0x7f6a080ab940 msgr2=0x7f6a080abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 1 -- 172.21.15.121:0/706925840 shutdown_connections 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 1 -- 172.21.15.121:0/706925840 wait complete. 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7fd1bd6bf640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7fd1bd6bf640 10 monclient: monmap: 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.887 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.864+0000 7f6a0e59b640 10 monclient: monmap: 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.889 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 1 Processor -- start 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 1 -- start start 2024-09-13T13:51:44.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 10 monclient: init 2024-09-13T13:51:44.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 1 Processor -- start 2024-09-13T13:51:44.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 1 -- start start 2024-09-13T13:51:44.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient: init 2024-09-13T13:51:44.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 10 monclient: _add_conns ranks=[4,3,1,8,7,6,2,0,5] 2024-09-13T13:51:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 10 monclient(hunting): picked mon.noname-c con 0x7fd1b809d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 10 monclient(hunting): picked mon.noname-g con 0x7fd1b80a6160 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient: _add_conns ranks=[8,6,7,4,0,1,3,2,5] 2024-09-13T13:51:44.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1b6ffd640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fd1b809d150 0x7fd1b81cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33700/0 (socket says 172.21.15.121:33700) 2024-09-13T13:51:44.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient(hunting): picked mon.noname-e con 0x7f6a0809d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:44.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1b6ffd640 1 -- 172.21.15.121:0/2050402201 learned_addr learned my addr 172.21.15.121:0/2050402201 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7fd1bd6bf640 10 monclient(hunting): picked mon.noname-f con 0x7fd1b80a6c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b8139e30 con 0x7fd1b80a6c50 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b8073890 con 0x7fd1b80a6160 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b80a45a0 con 0x7fd1b809d150 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7fd1bd6bf640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7fd1bd6bf640 10 monclient(hunting): authenticate will time out at 784.496147s 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient(hunting): picked mon.noname-d con 0x7f6a080a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient(hunting): picked mon.noname-i con 0x7f6a080a6c50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:44.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a080a8e90 con 0x7f6a080a6160 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a08124a30 con 0x7f6a080a6c50 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a08126d90 con 0x7f6a0809d150 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.865+0000 7f6a0e59b640 10 monclient(hunting): authenticate will time out at 784.495988s 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7f6a07fff640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f6a0809d150 0x7f6a081cd920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56542/0 (socket says 172.21.15.121:56542) 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.866+0000 7f6a07fff640 1 -- 172.21.15.121:0/645848434 learned_addr learned my addr 172.21.15.121:0/645848434 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd1b77fe640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.153:6789/0 conn(0x7fd1b80a6c50 legacy=0x7fd1b81da4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd1b6ffd640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.121:6791/0 conn(0x7fd1b809d150 legacy=0x7fd1b81cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd1b67fc640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.153:6790/0 conn(0x7fd1b80a6160 legacy=0x7fd1b81d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a07fff640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.121:6793/0 conn(0x7f6a0809d150 legacy=0x7f6a081cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068077564 0 0) 0x7fd1b0001800 con 0x7fd1b80a6c50 2024-09-13T13:51:44.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1336195730 0 0) 0x7f6a08126d90 con 0x7f6a0809d150 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f69fffff640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.121:6792/0 conn(0x7f6a080a6160 legacy=0x7f6a081393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): my global_id is 4274 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69e8003520 con 0x7f6a0809d150 2024-09-13T13:51:44.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): my global_id is 4294 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1a4003520 con 0x7fd1b80a6c50 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a0cb38640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.153:6792/0 conn(0x7f6a080a6c50 legacy=0x7f6a081d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163835648 0 0) 0x7fd1b80a45a0 con 0x7fd1b809d150 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 10 monclient(hunting): my global_id is 4234 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b0001800 con 0x7fd1b809d150 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799837781 0 0) 0x7f6a080a8e90 con 0x7f6a080a6160 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): my global_id is 4263 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a08126d90 con 0x7f6a080a6160 2024-09-13T13:51:44.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4034453810 0 0) 0x7f6a08124a30 con 0x7f6a080a6c50 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 10 monclient(hunting): my global_id is 4300 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a080a8e90 con 0x7f6a080a6c50 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.868+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 408698456 0 0) 0x7fd1a4003520 con 0x7fd1b80a6c50 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1b80a45a0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4141835120 0 0) 0x7fd1b0001800 con 0x7fd1b809d150 2024-09-13T13:51:44.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1a4003520 con 0x7fd1b809d150 2024-09-13T13:51:44.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2763781604 0 0) 0x7f69e8003520 con 0x7f6a0809d150 2024-09-13T13:51:44.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a08124a30 con 0x7f6a0809d150 2024-09-13T13:51:44.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801371893 0 0) 0x7fd1b8073890 con 0x7fd1b80a6160 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): my global_id is 4251 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b0001800 con 0x7fd1b80a6160 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd1b0003990 con 0x7fd1b80a6c50 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.902 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:44.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd1a8003610 con 0x7fd1b809d150 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.904 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 207493024 0 0) 0x7fd1b80a45a0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.121:6791/0 conn(0x7fd1b809d150 legacy=0x7fd1b81cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2660353545 0 0) 0x7f6a080a8e90 con 0x7f6a080a6c50 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69e8003520 con 0x7f6a080a6c50 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6a00003e80 con 0x7f6a080a6c50 2024-09-13T13:51:44.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): dump: 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.906 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1354225901 0 0) 0x7f6a08126d90 con 0x7f6a080a6160 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a080a8e90 con 0x7f6a080a6160 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f69f0003610 con 0x7f6a080a6160 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): dump: 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.907 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.908 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f69f8001a90 con 0x7f6a0809d150 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient(hunting): dump: 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.909 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 809622186 0 0) 0x7f6a08124a30 con 0x7f6a0809d150 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.153:6792/0 conn(0x7f6a080a6c50 legacy=0x7f6a081d5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.121:6792/0 conn(0x7f6a080a6160 legacy=0x7f6a081393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 monclient: found mon.e 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:44.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a081264f0 con 0x7f6a0809d150 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 5 monclient: authenticate success, global_id 4274 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 10 monclient: _renew_subs 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a08125590 con 0x7f6a0809d150 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 10 monclient: _renew_subs 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:44.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a08139cc0 con 0x7f6a0809d150 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.153:6790/0 conn(0x7fd1b80a6160 legacy=0x7fd1b81d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 monclient: found mon.f 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b81cf400 con 0x7fd1b80a6c50 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd19f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd1bd6bf640 5 monclient: authenticate success, global_id 4294 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd1bd6bf640 10 monclient: _renew_subs 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.869+0000 7fd1bd6bf640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:44.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b8071530 con 0x7fd1b80a6c50 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd1bd6bf640 10 monclient: _renew_subs 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd1bd6bf640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1b8073cb0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1b0003ee0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd19f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd1b0004450 con 0x7fd1b80a6c50 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd1b57fa640 4 set_mon_vals no callback set 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7f6a0e59b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:44.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6793/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f6a080a4230 con 0x7f6a0809d150 2024-09-13T13:51:44.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.870+0000 7fd19f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7fd19f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:44.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7fd19f7fe640 10 monclient: dump: 2024-09-13T13:51:44.915 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.916 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.917 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7fd1bd6bf640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:51:44.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.153:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fd198000f80 con 0x7fd1b80a6c50 2024-09-13T13:51:44.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fd1b00045f0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f69f8004dd0 con 0x7f6a0809d150 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f69f80052c0 con 0x7f6a0809d150 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 10 monclient: dump: 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f69f8011940 con 0x7f6a0809d150 2024-09-13T13:51:44.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a06ffd640 4 set_mon_vals no callback set 2024-09-13T13:51:44.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.871+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f69f804e340 con 0x7f6a0809d150 2024-09-13T13:51:44.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7fd1b004e460 con 0x7fd1b80a6c50 2024-09-13T13:51:44.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7fd1b6ffd640 10 monclient: get_auth_request con 0x7fd1a40401e0 auth_method 0 2024-09-13T13:51:44.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6804/2141781943 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69e804ae30 con 0x7f69e8047180 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7f69f804e6b0 con 0x7f6a0809d150 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7f6a057fa640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7f69fffff640 10 monclient: get_auth_request con 0x7f69e8040520 auth_method 0 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.121:6808/3060725185 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1a404abe0 con 0x7fd1a4046f30 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7fd1b004e8d0 con 0x7fd1b80a6c50 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7fd19f7fe640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-13T13:51:44.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7f6a0cb38640 10 monclient: get_auth_request con 0x7f69e8047180 auth_method 0 2024-09-13T13:51:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.872+0000 7fd1b67fc640 10 monclient: get_auth_request con 0x7fd1a4046f30 auth_method 0 2024-09-13T13:51:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.877+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== osd.0 v1:172.21.15.121:6808/3060725185 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fd1a404abe0 con 0x7fd1a4046f30 2024-09-13T13:51:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.877+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== osd.2 v1:172.21.15.121:6804/2141781943 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f69e804ae30 con 0x7f69e8047180 2024-09-13T13:51:44.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.896+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 --> v1:172.21.15.121:6808/3060725185 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd198002c70 con 0x7fd1a4046f30 2024-09-13T13:51:44.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.896+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 --> v1:172.21.15.121:6804/2141781943 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6a081c1ac0 con 0x7f69e8047180 2024-09-13T13:51:44.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.896+0000 7fd19f7fe640 1 -- 172.21.15.121:0/2050402201 <== osd.0 v1:172.21.15.121:6808/3060725185 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7fd198002c70 con 0x7fd1a4046f30 2024-09-13T13:51:44.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a057fa640 1 -- 172.21.15.121:0/645848434 <== osd.2 v1:172.21.15.121:6804/2141781943 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3325991546) 0x7f6a081c1ac0 con 0x7f69e8047180 2024-09-13T13:51:44.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f611380f640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f611380f640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.121:6808/3060725185 conn(0x7fd1a4046f30 legacy=0x7fd1a4049370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f611380f640 10 monclient: monmap: 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.899034+0000 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.899034+0000 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.925 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.153:6812/1206688214 conn(0x7fd1a40401e0 legacy=0x7fd1a40426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 10 monclient: shutdown 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 >> v1:172.21.15.153:6789/0 conn(0x7fd1b80a6c50 legacy=0x7fd1b81da4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1b77fe640 1 -- 172.21.15.121:0/2050402201 reap_dead start 2024-09-13T13:51:44.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 shutdown_connections 2024-09-13T13:51:44.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 >> 172.21.15.121:0/2050402201 conn(0x7fd1b80ab940 msgr2=0x7fd1b80aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.121:6804/2141781943 conn(0x7f69e8047180 legacy=0x7f69e80495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 shutdown_connections 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.153:6812/1206688214 conn(0x7f69e8040520 legacy=0x7f69e80429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 10 monclient: shutdown 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 >> v1:172.21.15.121:6793/0 conn(0x7f6a0809d150 legacy=0x7f6a081cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0cb38640 1 -- 172.21.15.121:0/645848434 reap_dead start 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 shutdown_connections 2024-09-13T13:51:44.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.897+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 >> 172.21.15.121:0/645848434 conn(0x7f6a080ab940 msgr2=0x7f6a081c1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7fa32014b640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7fa32014b640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7fa32014b640 10 monclient: monmap: 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.899790+0000 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.899790+0000 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.929 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 shutdown_connections 2024-09-13T13:51:44.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7fd1bd6bf640 1 -- 172.21.15.121:0/2050402201 wait complete. 2024-09-13T13:51:44.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7f611380f640 1 Processor -- start 2024-09-13T13:51:44.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7f6a0e59b640 1 -- 172.21.15.121:0/645848434 wait complete. 2024-09-13T13:51:44.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7f611380f640 1 -- start start 2024-09-13T13:51:44.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7f611380f640 10 monclient: init 2024-09-13T13:51:44.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.898+0000 7fa32014b640 1 Processor -- start 2024-09-13T13:51:44.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 1 -- start start 2024-09-13T13:51:44.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient: init 2024-09-13T13:51:44.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f611380f640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f611380f640 10 monclient: _add_conns ranks=[2,6,8,7,1,0,3,5,4] 2024-09-13T13:51:44.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f61115ab640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f610c0a3e30 0x7f610c073fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59118/0 (socket says 172.21.15.121:59118) 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f61115ab640 1 -- 172.21.15.121:0/3686532487 learned_addr learned my addr 172.21.15.121:0/3686532487 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f611380f640 10 monclient(hunting): picked mon.noname-b con 0x7f610c0a3e30 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f611380f640 10 monclient(hunting): picked mon.noname-d con 0x7f610c0a4210 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient: _add_conns ranks=[5,0,4,6,8,7,2,3,1] 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient(hunting): picked mon.noname-h con 0x7fa31806c8b0 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient(hunting): picked mon.noname-a con 0x7fa31809bbe0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient(hunting): picked mon.noname-c con 0x7fa31813f570 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3180a1080 con 0x7fa31809bbe0 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31806cfe0 con 0x7fa31813f570 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa31e6e8640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fa31813f570 0x7fa318141960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33716/0 (socket says 172.21.15.121:33716) 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa31e6e8640 1 -- 172.21.15.121:0/203645660 learned_addr learned my addr 172.21.15.121:0/203645660 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 1 -- 172.21.15.121:0/203645660 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31806d4a0 con 0x7fa31806c8b0 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7fa32014b640 10 monclient(hunting): authenticate will time out at 784.529964s 2024-09-13T13:51:44.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.899+0000 7f611380f640 10 monclient(hunting): picked mon.noname-e con 0x7f610c13d050 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:44.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.900+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c0737f0 con 0x7f610c0a3e30 2024-09-13T13:51:44.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.900+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c0734a0 con 0x7f610c0a4210 2024-09-13T13:51:44.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.900+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c0a5b60 con 0x7f610c13d050 2024-09-13T13:51:44.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.900+0000 7f611380f640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.900+0000 7f611380f640 10 monclient(hunting): authenticate will time out at 784.530320s 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.901+0000 7f7cc87d0640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.901+0000 7f7cc87d0640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.901+0000 7f7cc87d0640 10 monclient: monmap: 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.903260+0000 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.903260+0000 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.936 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.901+0000 7f61115ab640 1 -- 172.21.15.121:0/3686532487 >> v1:172.21.15.121:6790/0 conn(0x7f610c0a3e30 legacy=0x7f610c073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31e6e8640 1 -- 172.21.15.121:0/203645660 >> v1:172.21.15.121:6791/0 conn(0x7fa31813f570 legacy=0x7fa318141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f13895ab640 10 monclient: get_monmap_and_config 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f13895ab640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f13895ab640 10 monclient: monmap: 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:44.904050+0000 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:44.904050+0000 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.938 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.939 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685506313 0 0) 0x7fa31806cfe0 con 0x7fa31813f570 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31cee5640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31cee5640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31cee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31dee7640 1 -- 172.21.15.121:0/203645660 >> v1:172.21.15.153:6791/0 conn(0x7fa31806c8b0 legacy=0x7fa3180688e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31cee5640 10 monclient(hunting): my global_id is 4243 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa304003520 con 0x7fa31813f570 2024-09-13T13:51:44.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7fa31d6e6640 1 -- 172.21.15.121:0/203645660 >> v1:172.21.15.121:6789/0 conn(0x7fa31809bbe0 legacy=0x7fa318068ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6110daa640 1 -- 172.21.15.121:0/3686532487 >> v1:172.21.15.121:6792/0 conn(0x7f610c0a4210 legacy=0x7f610c0a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702508113 0 0) 0x7fa31806d4a0 con 0x7fa31806c8b0 2024-09-13T13:51:44.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): my global_id is 4262 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa31806cfe0 con 0x7fa31806c8b0 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509856584 0 0) 0x7fa3080013d0 con 0x7fa31809bbe0 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient(hunting): my global_id is 4320 2024-09-13T13:51:44.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa31806d4a0 con 0x7fa31809bbe0 2024-09-13T13:51:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6111dac640 1 -- 172.21.15.121:0/3686532487 >> v1:172.21.15.121:6793/0 conn(0x7f610c13d050 legacy=0x7f610c13f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617335657 0 0) 0x7f610c0a5b60 con 0x7f610c13d050 2024-09-13T13:51:44.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 799410882 0 0) 0x7fa304003520 con 0x7fa31813f570 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 >> v1:172.21.15.153:6791/0 conn(0x7fa31806c8b0 legacy=0x7fa3180688e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 >> v1:172.21.15.121:6789/0 conn(0x7fa31809bbe0 legacy=0x7fa318068ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 monclient: found mon.noname-c 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3180a00e0 con 0x7fa31813f570 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa310001e60 con 0x7fa31813f570 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: dump: 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.945 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa32014b640 5 monclient: authenticate success, global_id 4243 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa32014b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f13895ab640 1 Processor -- start 2024-09-13T13:51:44.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): my global_id is 4283 2024-09-13T13:51:44.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f13895ab640 1 -- start start 2024-09-13T13:51:44.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f13895ab640 10 monclient: init 2024-09-13T13:51:44.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f8003520 con 0x7f610c13d050 2024-09-13T13:51:44.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768819939 0 0) 0x7f610c0734a0 con 0x7f610c0a4210 2024-09-13T13:51:44.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 10 monclient(hunting): my global_id is 4272 2024-09-13T13:51:44.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f80048e0 con 0x7f610c0a4210 2024-09-13T13:51:44.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f7cc87d0640 1 Processor -- start 2024-09-13T13:51:44.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f7cc87d0640 1 -- start start 2024-09-13T13:51:44.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.902+0000 7f7cc87d0640 10 monclient: init 2024-09-13T13:51:44.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 10 monclient: _add_conns ranks=[8,7,1,3,6,5,2,0,4] 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc656c640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f7cc00a6160 0x7f7cc00a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56560/0 (socket says 172.21.15.121:56560) 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc656c640 1 -- 172.21.15.121:0/1991526687 learned_addr learned my addr 172.21.15.121:0/1991526687 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 10 monclient(hunting): picked mon.noname-e con 0x7f7cc00a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 10 monclient(hunting): picked mon.noname-i con 0x7f7cc009d150 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 10 monclient(hunting): picked mon.noname-f con 0x7f7cc00a6c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc009e210 con 0x7f7cc00a6c50 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc00a7ef0 con 0x7f7cc009d150 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa310004c20 con 0x7fa31813f570 2024-09-13T13:51:44.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc009d970 con 0x7f7cc00a6160 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f7cc87d0640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f7cc87d0640 10 monclient(hunting): authenticate will time out at 784.534017s 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 1 -- 172.21.15.121:0/203645660 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa310005190 con 0x7fa31813f570 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7fa31cee5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa31cee5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa31cee5640 10 monclient: dump: 2024-09-13T13:51:44.951 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.952 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.953 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.953 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa32014b640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa32014b640 4 set_mon_vals no callback set 2024-09-13T13:51:44.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2669539511 0 0) 0x7f60f80048e0 con 0x7f610c0a4210 2024-09-13T13:51:44.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 >> v1:172.21.15.121:6793/0 conn(0x7f610c13d050 legacy=0x7f610c13f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 >> v1:172.21.15.121:6790/0 conn(0x7f610c0a3e30 legacy=0x7f610c073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 1 monclient: found mon.noname-d 2024-09-13T13:51:44.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient: _add_conns ranks=[0,3,4,7,1,8,6,2,5] 2024-09-13T13:51:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c143b90 con 0x7f610c0a4210 2024-09-13T13:51:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient(hunting): picked mon.noname-a con 0x7f1384138bd0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient(hunting): picked mon.noname-g con 0x7f1384138fb0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:44.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient(hunting): picked mon.noname-c con 0x7f1384139420 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f611380f640 5 monclient: authenticate success, global_id 4272 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f611380f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f60fc001b10 con 0x7f610c0a4210 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f6103fff640 10 monclient: dump: 2024-09-13T13:51:44.956 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.957 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.903+0000 7f611380f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60fc003610 con 0x7f610c0a4210 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f6103fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f6103fff640 1 -- 172.21.15.121:0/3686532487 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f60fc001b10 con 0x7f610c0a4210 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f6103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 4 set_mon_vals no callback set 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f6103fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f6103fff640 10 monclient: dump: 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.958 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.959 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138409e100 con 0x7f1384138bd0 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138409db40 con 0x7f1384138fb0 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138409de20 con 0x7f1384139420 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13895ab640 10 monclient(hunting): authenticate will time out at 784.534618s 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13837fe640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f1384139420 0x7f1384141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33728/0 (socket says 172.21.15.121:33728) 2024-09-13T13:51:44.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f13837fe640 1 -- 172.21.15.121:0/293431254 learned_addr learned my addr 172.21.15.121:0/293431254 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 10 monclient: shutdown 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 >> v1:172.21.15.121:6792/0 conn(0x7f610c0a4210 legacy=0x7f610c0a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 shutdown_connections 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 >> 172.21.15.121:0/3686532487 conn(0x7f610c0a9430 msgr2=0x7f610c0a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa32014b640 10 monclient: shutdown 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa32014b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.904+0000 7fa32014b640 1 -- 172.21.15.121:0/203645660 >> v1:172.21.15.121:6791/0 conn(0x7fa31813f570 legacy=0x7fa318141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7fa32014b640 1 -- 172.21.15.121:0/203645660 shutdown_connections 2024-09-13T13:51:44.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7fa32014b640 1 -- 172.21.15.121:0/203645660 >> 172.21.15.121:0/203645660 conn(0x7fa3180a83b0 msgr2=0x7fa3180a87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 shutdown_connections 2024-09-13T13:51:44.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7f611380f640 1 -- 172.21.15.121:0/3686532487 wait complete. 2024-09-13T13:51:44.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7fa32014b640 1 -- 172.21.15.121:0/203645660 shutdown_connections 2024-09-13T13:51:44.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7fa32014b640 1 -- 172.21.15.121:0/203645660 wait complete. 2024-09-13T13:51:44.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7fa32014b640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7f611380f640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7f7cc656c640 1 -- 172.21.15.121:0/1991526687 >> v1:172.21.15.121:6793/0 conn(0x7f7cc00a6160 legacy=0x7f7cc00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7f611380f640 10 monclient: monmap: 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.964 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.905+0000 7fa32014b640 10 monclient: monmap: 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.965 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328401487 0 0) 0x7f7cc009d970 con 0x7f7cc00a6160 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient(hunting): my global_id is 4292 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cac003520 con 0x7f7cc00a6160 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7fa32014b640 1 Processor -- start 2024-09-13T13:51:44.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc6d6d640 1 -- 172.21.15.121:0/1991526687 >> v1:172.21.15.153:6789/0 conn(0x7f7cc00a6c50 legacy=0x7f7cc0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc5d6b640 1 -- 172.21.15.121:0/1991526687 >> v1:172.21.15.153:6792/0 conn(0x7f7cc009d150 legacy=0x7f7cc013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f611380f640 1 Processor -- start 2024-09-13T13:51:44.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1727049071 0 0) 0x7f7cac003520 con 0x7f7cc00a6160 2024-09-13T13:51:44.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 >> v1:172.21.15.153:6792/0 conn(0x7f7cc009d150 legacy=0x7f7cc013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 >> v1:172.21.15.153:6789/0 conn(0x7f7cc00a6c50 legacy=0x7f7cc0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 monclient: found mon.noname-e 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc009e630 con 0x7f7cc00a6160 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7cbc001ba0 con 0x7f7cc00a6160 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: dump: 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.969 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.970 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc87d0640 5 monclient: authenticate success, global_id 4292 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc87d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7fa32014b640 1 -- start start 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7fa32014b640 10 monclient: init 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7cbc0048a0 con 0x7f7cc00a6160 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc87d0640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc87d0640 4 set_mon_vals no callback set 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 1 -- 172.21.15.121:0/1991526687 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7cbc004ed0 con 0x7f7cc00a6160 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f7cc556a640 10 monclient: dump: 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.972 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.973 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f611380f640 1 -- start start 2024-09-13T13:51:44.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f611380f640 10 monclient: init 2024-09-13T13:51:44.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f13827fc640 1 -- 172.21.15.121:0/293431254 >> v1:172.21.15.153:6790/0 conn(0x7f1384138fb0 legacy=0x7f138413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f611380f640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7f611380f640 10 monclient: _add_conns ranks=[6,0,5,7,2,8,4,1,3] 2024-09-13T13:51:44.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1382ffd640 1 -- 172.21.15.121:0/293431254 >> v1:172.21.15.121:6789/0 conn(0x7f1384138bd0 legacy=0x7f13840a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f13837fe640 1 -- 172.21.15.121:0/293431254 >> v1:172.21.15.121:6791/0 conn(0x7f1384139420 legacy=0x7f1384141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7fa32014b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 10 monclient(hunting): picked mon.noname-d con 0x7f610c0a3e30 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:44.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 10 monclient(hunting): picked mon.noname-a con 0x7f610c0a4210 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:44.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 10 monclient(hunting): picked mon.noname-h con 0x7f610c13d050 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c1410b0 con 0x7f610c0a4210 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7fa32014b640 10 monclient: _add_conns ranks=[7,5,8,0,2,4,1,6,3] 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.906+0000 7fa32014b640 10 monclient(hunting): picked mon.noname-i con 0x7fa31806c8b0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa32014b640 10 monclient(hunting): picked mon.noname-h con 0x7fa31809bbe0 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c1438b0 con 0x7f610c13d050 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c1429b0 con 0x7f610c0a3e30 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f611380f640 10 monclient(hunting): authenticate will time out at 784.537326s 2024-09-13T13:51:44.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa31d6e6640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7fa31809bbe0 0x7fa3180906f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:45066/0 (socket says 172.21.15.121:45066) 2024-09-13T13:51:44.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f6111dac640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f610c13d050 0x7f610c1d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:45068/0 (socket says 172.21.15.121:45068) 2024-09-13T13:51:44.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f6111dac640 1 -- 172.21.15.121:0/2289301162 learned_addr learned my addr 172.21.15.121:0/2289301162 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa31dee7640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7fa31806c8b0 0x7fa31808ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.121:60706/0 (socket says 172.21.15.121:60706) 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2952064566 0 0) 0x7f138409db40 con 0x7f1384138fb0 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): my global_id is 4260 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f136c003520 con 0x7f1384138fb0 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2936990964 0 0) 0x7f138409e100 con 0x7f1384138bd0 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient(hunting): my global_id is 4329 2024-09-13T13:51:44.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f138409db40 con 0x7f1384138bd0 2024-09-13T13:51:44.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa31e6e8640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 0x7fa3181db000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56572/0 (socket says 172.21.15.121:56572) 2024-09-13T13:51:44.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f7cc87d0640 10 monclient: shutdown 2024-09-13T13:51:44.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f7cc87d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 >> v1:172.21.15.121:6793/0 conn(0x7f7cc00a6160 legacy=0x7f7cc00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4079349132 0 0) 0x7f136c003520 con 0x7f1384138fb0 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 >> v1:172.21.15.121:6791/0 conn(0x7f1384139420 legacy=0x7f1384141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 >> v1:172.21.15.121:6789/0 conn(0x7f1384138bd0 legacy=0x7f13840a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 monclient: found mon.noname-g 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13840a5da0 con 0x7f1384138fb0 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:44.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1374001b30 con 0x7f1384138fb0 2024-09-13T13:51:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f13895ab640 5 monclient: authenticate success, global_id 4260 2024-09-13T13:51:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f13895ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa31e6e8640 1 -- 172.21.15.121:0/335889331 learned_addr learned my addr 172.21.15.121:0/335889331 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:44.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 10 monclient: dump: 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.985 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f13895ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa32014b640 10 monclient(hunting): picked mon.noname-e con 0x7fa31813f570 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa32014b640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31809e9c0 con 0x7fa31809bbe0 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa32014b640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa318142c10 con 0x7fa31806c8b0 2024-09-13T13:51:44.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1374003610 con 0x7f1384138fb0 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f1381ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f1381ffb640 1 -- 172.21.15.121:0/293431254 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1374003b80 con 0x7f1384138fb0 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f1381ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.907+0000 7fa32014b640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3180918d0 con 0x7fa31813f570 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f1381ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f1381ffb640 10 monclient: dump: 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:44.987 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:44.988 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:44.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 4 set_mon_vals no callback set 2024-09-13T13:51:44.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7fa32014b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7fa32014b640 10 monclient(hunting): authenticate will time out at 784.538042s 2024-09-13T13:51:44.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 shutdown_connections 2024-09-13T13:51:44.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 >> 172.21.15.121:0/1991526687 conn(0x7f7cc00ab940 msgr2=0x7f7cc00abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 shutdown_connections 2024-09-13T13:51:44.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 10 monclient: shutdown 2024-09-13T13:51:44.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:44.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 1 -- 172.21.15.121:0/293431254 >> v1:172.21.15.153:6790/0 conn(0x7f1384138fb0 legacy=0x7f138413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:44.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f7cc87d0640 1 -- 172.21.15.121:0/1991526687 wait complete. 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f7cc87d0640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f7cc87d0640 10 monclient: monmap: 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.991 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 1 -- 172.21.15.121:0/293431254 shutdown_connections 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 1 -- 172.21.15.121:0/293431254 >> 172.21.15.121:0/293431254 conn(0x7f13840ab800 msgr2=0x7f13840abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:44.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 1 -- 172.21.15.121:0/293431254 shutdown_connections 2024-09-13T13:51:44.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.908+0000 7f13895ab640 1 -- 172.21.15.121:0/293431254 wait complete. 2024-09-13T13:51:44.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 1 Processor -- start 2024-09-13T13:51:44.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 1 -- start start 2024-09-13T13:51:44.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient: init 2024-09-13T13:51:44.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6111dac640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.153:6791/0 conn(0x7f610c13d050 legacy=0x7f610c1d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f13895ab640 10 monclient: build_initial_monmap 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f13895ab640 10 monclient: monmap: 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:44.995 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient: _add_conns ranks=[2,7,5,1,0,4,6,8,3] 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc009d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient(hunting): picked mon.noname-i con 0x7f7cc00a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient(hunting): picked mon.noname-h con 0x7f7cc00a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f61115ab640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.121:6792/0 conn(0x7f610c0a3e30 legacy=0x7f610c13a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6110daa640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.121:6789/0 conn(0x7f610c0a4210 legacy=0x7f610c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995026954 0 0) 0x7f610c1429b0 con 0x7f610c0a3e30 2024-09-13T13:51:44.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6101ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7fa31e6e8640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 legacy=0x7fa3181db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7fa31dee7640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.153:6792/0 conn(0x7fa31806c8b0 legacy=0x7fa31808ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7fa31d6e6640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.153:6791/0 conn(0x7fa31809bbe0 legacy=0x7fa3180906f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6101ffb640 10 monclient(hunting): my global_id is 4281 2024-09-13T13:51:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60f0003520 con 0x7f610c0a3e30 2024-09-13T13:51:44.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc0136840 con 0x7f7cc009d150 2024-09-13T13:51:44.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc0135bc0 con 0x7f7cc00a6c50 2024-09-13T13:51:44.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc0136b20 con 0x7f7cc00a6160 2024-09-13T13:51:44.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.909+0000 7f7cc87d0640 10 monclient(hunting): authenticate will time out at 784.539859s 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f7cc6d6d640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f7cc00a6c50 0x7f7cc01e2830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:45084/0 (socket says 172.21.15.121:45084) 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f7cc6d6d640 1 -- 172.21.15.121:0/870074930 learned_addr learned my addr 172.21.15.121:0/870074930 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2817439436 0 0) 0x7fa318142c10 con 0x7fa31806c8b0 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:44.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): my global_id is 4318 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2fc003550 con 0x7fa31806c8b0 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860011020 0 0) 0x7fa31809e9c0 con 0x7fa31809bbe0 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): my global_id is 4280 2024-09-13T13:51:45.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa318142c10 con 0x7fa31809bbe0 2024-09-13T13:51:45.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 275014202 0 0) 0x7f610c1410b0 con 0x7f610c0a4210 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): my global_id is 4338 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f610c1429b0 con 0x7f610c0a4210 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1347201655 0 0) 0x7f60f8003520 con 0x7f610c13d050 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): my global_id is 4271 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f610c1410b0 con 0x7f610c13d050 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3515600536 0 0) 0x7f60f0003520 con 0x7f610c0a3e30 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60f8003520 con 0x7f610c0a3e30 2024-09-13T13:51:45.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6108001c40 con 0x7f610c0a3e30 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3850962309 0 0) 0x7fa3180918d0 con 0x7fa31813f570 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): dump: 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.003 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.004 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 10 monclient(hunting): my global_id is 4301 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa31809e9c0 con 0x7fa31813f570 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 890174391 0 0) 0x7f610c1429b0 con 0x7f610c0a4210 2024-09-13T13:51:45.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60f0003520 con 0x7f610c0a4210 2024-09-13T13:51:45.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f13895ab640 1 Processor -- start 2024-09-13T13:51:45.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f60fc001b10 con 0x7f610c0a4210 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f6101ffb640 10 monclient(hunting): dump: 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.007 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f13895ab640 1 -- start start 2024-09-13T13:51:45.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.910+0000 7f13895ab640 10 monclient: init 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 108026750 0 0) 0x7f60f8003520 con 0x7f610c0a3e30 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.153:6791/0 conn(0x7f610c13d050 legacy=0x7f610c1d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.121:6789/0 conn(0x7f610c0a4210 legacy=0x7f610c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 monclient: found mon.d 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c142c90 con 0x7f610c0a3e30 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6108004b00 con 0x7f610c0a3e30 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6108005070 con 0x7f610c0a3e30 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1787546960 0 0) 0x7fa31809e9c0 con 0x7fa31813f570 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3180918d0 con 0x7fa31813f570 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa3100020b0 con 0x7fa31813f570 2024-09-13T13:51:45.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.011 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f6101ffb640 10 monclient: dump: 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.012 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f611380f640 5 monclient: authenticate success, global_id 4281 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f611380f640 10 monclient: _renew_subs 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f611380f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f610c13b410 con 0x7f610c0a3e30 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f61037fe640 4 set_mon_vals no callback set 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient: _add_conns ranks=[1,4,0,5,6,7,2,8,3] 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient(hunting): picked mon.noname-f con 0x7f1384138bd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient(hunting): picked mon.noname-c con 0x7f1384138fb0 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient(hunting): picked mon.noname-a con 0x7f1384139420 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1384073060 con 0x7f1384139420 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f138409eee0 con 0x7f1384138bd0 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13840a7a60 con 0x7f1384138fb0 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7f13895ab640 10 monclient(hunting): authenticate will time out at 784.541866s 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f611380f640 10 monclient: _renew_subs 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f611380f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f610c076ff0 con 0x7f610c0a3e30 2024-09-13T13:51:45.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 682003124 0 0) 0x7fa3180918d0 con 0x7fa31813f570 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.153:6792/0 conn(0x7fa31806c8b0 legacy=0x7fa31808ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.153:6791/0 conn(0x7fa31809bbe0 legacy=0x7fa3180906f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 monclient: found mon.e 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa318072a50 con 0x7fa31813f570 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa32014b640 5 monclient: authenticate success, global_id 4301 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa32014b640 10 monclient: _renew_subs 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa32014b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:45.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.911+0000 7fa32014b640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3180721b0 con 0x7fa31813f570 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa32014b640 10 monclient: _renew_subs 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa32014b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa32014b640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa318074910 con 0x7fa31813f570 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi121.stdout:51539607557 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cc656c640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.121:6790/0 conn(0x7f7cc009d150 legacy=0x7f7cc01da4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cc5d6b640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.153:6792/0 conn(0x7f7cc00a6160 legacy=0x7f7cc0139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f13837fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1384139420 0x7f13841dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45462/0 (socket says 172.21.15.121:45462) 2024-09-13T13:51:45.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f13837fe640 1 -- 172.21.15.121:0/2048584876 learned_addr learned my addr 172.21.15.121:0/2048584876 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa31e6e8640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 legacy=0x7fa3181db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T13:51:45.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa31e6e8640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 legacy=0x7fa3181db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:45.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa31e6e8640 1 --1- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 0x7fa3181db000 :-1 s=OPENED pgs=62 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:45.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa31e6e8640 1 --1- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 0x7fa3181db000 :-1 s=OPENED pgs=62 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.121:6793/0 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6793/0 conn(0x7fa31813f570 legacy=0x7fa3181db000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient: _add_conns ranks=[4,6,0,1,2,5,3,8,7] 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient(hunting): picked mon.c con 0x7fa2fc004160 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient(hunting): picked mon.d con 0x7fa2fc006150 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient(hunting): picked mon.a con 0x7fa2fc00d320 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa3180918d0 con 0x7fa2fc00d320 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa31809e9c0 con 0x7fa2fc004160 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa2fc013490 con 0x7fa2fc006150 2024-09-13T13:51:45.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7fa30e7fc640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923409551 0 0) 0x7f7cc0136b20 con 0x7f7cc00a6160 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 10 monclient(hunting): my global_id is 4327 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ca4003520 con 0x7f7cc00a6160 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.912+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589979160 0 0) 0x7f7cc0136840 con 0x7f7cc009d150 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): my global_id is 4259 2024-09-13T13:51:45.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc0136b20 con 0x7f7cc009d150 2024-09-13T13:51:45.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cc6d6d640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.153:6791/0 conn(0x7f7cc00a6c50 legacy=0x7f7cc01e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f61080116f0 con 0x7f610c0a3e30 2024-09-13T13:51:45.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f611380f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.121:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f610c141c10 con 0x7f610c0a3e30 2024-09-13T13:51:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1288006160 0 0) 0x7f7ca4003520 con 0x7f7cc00a6160 2024-09-13T13:51:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7cc0136840 con 0x7f7cc00a6160 2024-09-13T13:51:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 729200236 0 0) 0x7f7cc0136b20 con 0x7f7cc009d150 2024-09-13T13:51:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7ca4003520 con 0x7f7cc009d150 2024-09-13T13:51:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7cb0001d10 con 0x7f7cc00a6160 2024-09-13T13:51:45.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f610804e000 con 0x7f610c0a3e30 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.024 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95075664 0 0) 0x7f7cc0135bc0 con 0x7f7cc00a6c50 2024-09-13T13:51:45.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): my global_id is 4289 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc0136b20 con 0x7f7cc00a6c50 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7cbc001b60 con 0x7f7cc009d150 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.026 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.153:6800/1083052486 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60f004ab30 con 0x7f60f0046e80 2024-09-13T13:51:45.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7f610804e4c0 con 0x7f610c0a3e30 2024-09-13T13:51:45.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6101ffb640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-13T13:51:45.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6110daa640 10 monclient: get_auth_request con 0x7f60f0040270 auth_method 0 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3851777720 0 0) 0x7f7ca4003520 con 0x7f7cc009d150 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.153:6792/0 conn(0x7f7cc00a6160 legacy=0x7f7cc0139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.153:6791/0 conn(0x7f7cc00a6c50 legacy=0x7f7cc01e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 monclient: found mon.b 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc0135380 con 0x7f7cc009d150 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cc87d0640 5 monclient: authenticate success, global_id 4259 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cc87d0640 10 monclient: _renew_subs 2024-09-13T13:51:45.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cc87d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc00a8f90 con 0x7f7cc009d150 2024-09-13T13:51:45.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7cbc001d00 con 0x7f7cc009d150 2024-09-13T13:51:45.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f6111dac640 10 monclient: get_auth_request con 0x7f60f0046e80 auth_method 0 2024-09-13T13:51:45.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7cbc0054d0 con 0x7f7cc009d150 2024-09-13T13:51:45.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cb6ffd640 10 monclient: dump: 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.031 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.913+0000 7f7cc4d69640 4 set_mon_vals no callback set 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f7cc87d0640 10 monclient: _renew_subs 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f7cc87d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7cc0139a50 con 0x7f7cc009d150 2024-09-13T13:51:45.033 INFO:teuthology.orchestra.run.smithi121.stdout:47244640262 2024-09-13T13:51:45.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T13:51:45.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f13827fc640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.121:6791/0 conn(0x7f1384138fb0 legacy=0x7f13841d7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f13837fe640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.121:6789/0 conn(0x7f1384139420 legacy=0x7f13841dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7fa31dee7640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6791/0 conn(0x7fa2fc004160 legacy=0x7fa2fc005a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720730278 0 0) 0x7f1384073060 con 0x7f1384139420 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): my global_id is 4347 2024-09-13T13:51:45.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1370003520 con 0x7f1384139420 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914134345 0 0) 0x7f13840a7a60 con 0x7f1384138fb0 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 10 monclient(hunting): my global_id is 4261 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1384073060 con 0x7f1384138fb0 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1155623273 0 0) 0x7fa31809e9c0 con 0x7fa2fc004160 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7fa30e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7fa30e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7fa30e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-13T13:51:45.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.914+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fa2fc0123c0 con 0x7fa2fc004160 2024-09-13T13:51:45.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cc87d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f7cc00a8ab0 con 0x7f7cc009d150 2024-09-13T13:51:45.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa31d6e6640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6792/0 conn(0x7fa2fc006150 legacy=0x7fa2fc00bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f1382ffd640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.153:6789/0 conn(0x7f1384138bd0 legacy=0x7f138412f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa31e6e8640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6789/0 conn(0x7fa2fc00d320 legacy=0x7fa2fc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f7cbc011b50 con 0x7f7cc009d150 2024-09-13T13:51:45.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1496267107 0 0) 0x7f1384073060 con 0x7f1384138fb0 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13840a7a60 con 0x7f1384138fb0 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1374003b00 con 0x7f1384138fb0 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.038 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 1485046942 0 0) 0x7fa2fc0123c0 con 0x7fa2fc004160 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6792/0 conn(0x7fa2fc006150 legacy=0x7fa2fc00bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6789/0 conn(0x7fa2fc00d320 legacy=0x7fa2fc00f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 monclient: found mon.c 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa31e6e8640 1 -- 172.21.15.121:0/335889331 reap_dead start 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fa2fc0148b0 con 0x7fa2fc004160 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fa318090e00 con 0x7fa2fc004160 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- mon_get_version(what=osdmap handle=2) -- 0x7fa3100020b0 con 0x7fa2fc004160 2024-09-13T13:51:45.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3926115254 0 0) 0x7f1370003520 con 0x7f1384139420 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1384073060 con 0x7f1384139420 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1655422789 0 0) 0x7f138409eee0 con 0x7f1384138bd0 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient(hunting): my global_id is 4312 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1370003520 con 0x7f1384138bd0 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- mon_get_version(what=osdmap handle=3) -- 0x7fa2fc0126d0 con 0x7fa2fc004160 2024-09-13T13:51:45.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.121:6791/0 -- mon_get_version(what=osdmap handle=4) -- 0x7fa2fc012880 con 0x7fa2fc004160 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa314003610 con 0x7fa2fc004160 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: dump: 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.042 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.043 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1761370418 0 0) 0x7f13840a7a60 con 0x7f1384138fb0 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.153:6789/0 conn(0x7f1384138bd0 legacy=0x7f138412f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.121:6789/0 conn(0x7f1384139420 legacy=0x7f13841dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 monclient: found mon.c 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13840a7510 con 0x7f1384138fb0 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f136b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f7cbc04e150 con 0x7f7cc009d150 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 5 monclient: authenticate success, global_id 4261 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 10 monclient: _renew_subs 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13840a5a60 con 0x7f1384138fb0 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 10 monclient: _renew_subs 2024-09-13T13:51:45.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f138409e620 con 0x7f1384138fb0 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa2fc012880 con 0x7fa2fc004160 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7fa30ffff640 4 set_mon_vals no callback set 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6800/1651307734 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cbc0048a0 con 0x7f7ca4046df0 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7f7cbc04e5c0 con 0x7f7cc009d150 2024-09-13T13:51:45.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cb6ffd640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-13T13:51:45.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.915+0000 7f7cc6d6d640 10 monclient: get_auth_request con 0x7f7ca4046df0 auth_method 0 2024-09-13T13:51:45.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1374001370 con 0x7f1384138fb0 2024-09-13T13:51:45.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f136b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f7cc5d6b640 10 monclient: get_auth_request con 0x7f7ca4040260 auth_method 0 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f13740015d0 con 0x7f1384138fb0 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f136b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f136b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f136b7fe640 10 monclient: dump: 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.048 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fa314003830 con 0x7fa2fc004160 2024-09-13T13:51:45.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f13895ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.121:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f13840abf60 con 0x7f1384138fb0 2024-09-13T13:51:45.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.916+0000 7f13817fa640 4 set_mon_vals no callback set 2024-09-13T13:51:45.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1374011ca0 con 0x7f1384138fb0 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 6 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7fa31404d840 con 0x7fa2fc004160 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.153:6804/3215471673 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2fc050490 con 0x7fa2fc04c800 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7fa31404db80 con 0x7fa2fc004160 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 8 ==== mon_get_version_reply(handle=2 version=20) ==== 24+0+0 (unknown 3899460254 0 0) 0x7fa31404de40 con 0x7fa2fc004160 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 10 monclient: handle_get_version_reply finishing 2 version 20 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_get_version_reply(handle=3 version=20) ==== 24+0+0 (unknown 1378011152 0 0) 0x7fa31404e100 con 0x7fa2fc004160 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 10 monclient: handle_get_version_reply finishing 3 version 20 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== mon.4 v1:172.21.15.121:6791/0 10 ==== mon_get_version_reply(handle=4 version=20) ==== 24+0+0 (unknown 2079923881 0 0) 0x7fa31404e3f0 con 0x7fa2fc004160 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa30e7fc640 10 monclient: handle_get_version_reply finishing 4 version 20 2024-09-13T13:51:45.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7fa31e6e8640 10 monclient: get_auth_request con 0x7fa2fc04c800 auth_method 0 2024-09-13T13:51:45.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f137404e320 con 0x7f1384138fb0 2024-09-13T13:51:45.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.153:6808/934815702 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f137004a700 con 0x7f1370046a50 2024-09-13T13:51:45.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7f137404e790 con 0x7f1384138fb0 2024-09-13T13:51:45.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7f136b7fe640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-13T13:51:45.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.917+0000 7f13837fe640 10 monclient: get_auth_request con 0x7f1370046a50 auth_method 0 2024-09-13T13:51:45.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.918+0000 7fa31d6e6640 10 monclient: get_auth_request con 0x7fa2fc045c40 auth_method 0 2024-09-13T13:51:45.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.918+0000 7f1382ffd640 10 monclient: get_auth_request con 0x7f137003fe50 auth_method 0 2024-09-13T13:51:45.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.919+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== osd.4 v1:172.21.15.153:6800/1083052486 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f60f004ab30 con 0x7f60f0046e80 2024-09-13T13:51:45.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.920+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== osd.1 v1:172.21.15.121:6800/1651307734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7cbc0048a0 con 0x7f7ca4046df0 2024-09-13T13:51:45.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.922+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== osd.5 v1:172.21.15.153:6804/3215471673 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa2fc050490 con 0x7fa2fc04c800 2024-09-13T13:51:45.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.922+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== osd.3 v1:172.21.15.153:6808/934815702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f137004a700 con 0x7f1370046a50 2024-09-13T13:51:45.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 --> v1:172.21.15.153:6800/1083052486 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f610c06dc10 con 0x7f60f0046e80 2024-09-13T13:51:45.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f6101ffb640 1 -- 172.21.15.121:0/2289301162 <== osd.4 v1:172.21.15.153:6800/1083052486 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3685469309) 0x7f610c06dc10 con 0x7f60f0046e80 2024-09-13T13:51:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.153:6800/1083052486 conn(0x7f60f0046e80 legacy=0x7f60f00492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.153:6812/1206688214 conn(0x7f60f0040270 legacy=0x7f60f0042730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 10 monclient: shutdown 2024-09-13T13:51:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 >> v1:172.21.15.121:6792/0 conn(0x7f610c0a3e30 legacy=0x7f610c13a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f6111dac640 1 -- 172.21.15.121:0/2289301162 reap_dead start 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 shutdown_connections 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.933+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 >> 172.21.15.121:0/2289301162 conn(0x7f610c0a9430 msgr2=0x7f610c06e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.934+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 shutdown_connections 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.935+0000 7f611380f640 1 -- 172.21.15.121:0/2289301162 wait complete. 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.939+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 --> v1:172.21.15.153:6808/934815702 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f138412fb50 con 0x7f1370046a50 2024-09-13T13:51:45.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f136b7fe640 1 -- 172.21.15.121:0/2048584876 <== osd.3 v1:172.21.15.153:6808/934815702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f138412fb50 con 0x7f1370046a50 2024-09-13T13:51:45.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.153:6808/934815702 conn(0x7f1370046a50 legacy=0x7f1370048e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.153:6812/1206688214 conn(0x7f137003fe50 legacy=0x7f1370042310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 10 monclient: shutdown 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 >> v1:172.21.15.121:6791/0 conn(0x7f1384138fb0 legacy=0x7f13841d7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13837fe640 1 -- 172.21.15.121:0/2048584876 reap_dead start 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 shutdown_connections 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 >> 172.21.15.121:0/2048584876 conn(0x7f13840ab800 msgr2=0x7f1384142430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.940+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 shutdown_connections 2024-09-13T13:51:45.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.941+0000 7f13895ab640 1 -- 172.21.15.121:0/2048584876 wait complete. 2024-09-13T13:51:45.059 INFO:teuthology.orchestra.run.smithi121.stdout:55834574853 2024-09-13T13:51:45.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T13:51:45.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.949+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 --> v1:172.21.15.121:6800/1651307734 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7cc0073a10 con 0x7f7ca4046df0 2024-09-13T13:51:45.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.949+0000 7f7cb6ffd640 1 -- 172.21.15.121:0/870074930 <== osd.1 v1:172.21.15.121:6800/1651307734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 2451364910) 0x7f7cc0073a10 con 0x7f7ca4046df0 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.121:6800/1651307734 conn(0x7f7ca4046df0 legacy=0x7f7ca4049230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.153:6812/1206688214 conn(0x7f7ca4040260 legacy=0x7f7ca4042720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 10 monclient: shutdown 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 >> v1:172.21.15.121:6790/0 conn(0x7f7cc009d150 legacy=0x7f7cc01da4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc6d6d640 1 -- 172.21.15.121:0/870074930 reap_dead start 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 shutdown_connections 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 >> 172.21.15.121:0/870074930 conn(0x7f7cc00ab940 msgr2=0x7f7cc00730b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 shutdown_connections 2024-09-13T13:51:45.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.950+0000 7f7cc87d0640 1 -- 172.21.15.121:0/870074930 wait complete. 2024-09-13T13:51:45.062 INFO:teuthology.orchestra.run.smithi121.stdout:51539607556 2024-09-13T13:51:45.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T13:51:45.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.951+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 --> v1:172.21.15.153:6804/3215471673 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa318072600 con 0x7fa2fc04c800 2024-09-13T13:51:45.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa30e7fc640 1 -- 172.21.15.121:0/335889331 <== osd.5 v1:172.21.15.153:6804/3215471673 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 982563978) 0x7fa318072600 con 0x7fa2fc04c800 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.153:6804/3215471673 conn(0x7fa2fc04c800 legacy=0x7fa2fc04ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.153:6812/1206688214 conn(0x7fa2fc045c40 legacy=0x7fa2fc048100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 10 monclient: shutdown 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 >> v1:172.21.15.121:6791/0 conn(0x7fa2fc004160 legacy=0x7fa2fc005a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 shutdown_connections 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.952+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 >> 172.21.15.121:0/335889331 conn(0x7fa3180a83b0 msgr2=0x7fa31809ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.953+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 shutdown_connections 2024-09-13T13:51:45.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:44.953+0000 7fa303fff640 1 -- 172.21.15.121:0/335889331 wait complete. 2024-09-13T13:51:45.065 INFO:teuthology.orchestra.run.smithi121.stdout:60129542148 2024-09-13T13:51:45.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T13:51:45.066 INFO:teuthology.orchestra.run.smithi121.stdout:55834574851 2024-09-13T13:51:45.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T13:51:45.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.180+0000 7f2713fa3640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.180+0000 7f2713fa3640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.180+0000 7f2713fa3640 10 monclient: monmap: 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.182476+0000 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.182476+0000 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.183 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.184 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.184 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.184 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.184 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.181+0000 7f2713fa3640 1 Processor -- start 2024-09-13T13:51:45.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.181+0000 7f2713fa3640 1 -- start start 2024-09-13T13:51:45.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.181+0000 7f2713fa3640 10 monclient: init 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient: _add_conns ranks=[8,6,2,3,1,0,7,5,4] 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient(hunting): picked mon.noname-e con 0x7f270c0a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient(hunting): picked mon.noname-d con 0x7f270c09d150 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f270c0a6c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c09e210 con 0x7f270c0a6c50 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c0a7ef0 con 0x7f270c09d150 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c09d970 con 0x7f270c0a6160 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f2713fa3640 10 monclient(hunting): authenticate will time out at 784.812538s 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f271153e640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f270c09d150 0x7f270c13e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52298/0 (socket says 172.21.15.121:52298) 2024-09-13T13:51:45.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.182+0000 7f271153e640 1 -- 172.21.15.121:0/70653269 learned_addr learned my addr 172.21.15.121:0/70653269 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.183+0000 7eff77fec640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.183+0000 7eff77fec640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.183+0000 7eff77fec640 10 monclient: monmap: 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.184789+0000 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.184789+0000 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.188 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.183+0000 7eff77fec640 1 Processor -- start 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.183+0000 7eff77fec640 1 -- start start 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.183+0000 7eff77fec640 10 monclient: init 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient: _add_conns ranks=[0,8,4,3,1,6,2,7,5] 2024-09-13T13:51:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient(hunting): picked mon.noname-a con 0x7eff700a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient(hunting): picked mon.noname-e con 0x7eff7009d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient(hunting): picked mon.noname-c con 0x7eff700a6c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7009e210 con 0x7eff700a6160 2024-09-13T13:51:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff700a7ef0 con 0x7eff700a6c50 2024-09-13T13:51:45.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7f271153e640 1 -- 172.21.15.121:0/70653269 >> v1:172.21.15.121:6792/0 conn(0x7f270c09d150 legacy=0x7f270c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7009d970 con 0x7eff7009d150 2024-09-13T13:51:45.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff77fec640 10 monclient(hunting): authenticate will time out at 784.814467s 2024-09-13T13:51:45.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff75d88640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7eff700a6160 0x7eff700a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45478/0 (socket says 172.21.15.121:45478) 2024-09-13T13:51:45.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7eff75d88640 1 -- 172.21.15.121:0/351390510 learned_addr learned my addr 172.21.15.121:0/351390510 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7f2711d3f640 1 -- 172.21.15.121:0/70653269 >> v1:172.21.15.121:6793/0 conn(0x7f270c0a6160 legacy=0x7f270c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.184+0000 7f2712540640 1 -- 172.21.15.121:0/70653269 >> v1:172.21.15.121:6790/0 conn(0x7f270c0a6c50 legacy=0x7f270c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1961816020 0 0) 0x7f270c09e210 con 0x7f270c0a6c50 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): my global_id is 4268 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f8003520 con 0x7f270c0a6c50 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 6657608 0 0) 0x7f270c09d970 con 0x7f270c0a6160 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): my global_id is 4310 2024-09-13T13:51:45.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270c09e210 con 0x7f270c0a6160 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 540003065 0 0) 0x7f270c0a7ef0 con 0x7f270c09d150 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient(hunting): my global_id is 4290 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270c09d970 con 0x7f270c09d150 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 979795161 0 0) 0x7f270c09e210 con 0x7f270c0a6160 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 >> v1:172.21.15.121:6792/0 conn(0x7f270c09d150 legacy=0x7f270c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 >> v1:172.21.15.121:6790/0 conn(0x7f270c0a6c50 legacy=0x7f270c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 monclient: found mon.noname-e 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:51:45.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c09e630 con 0x7f270c0a6160 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2708001bd0 con 0x7f270c0a6160 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: dump: 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.196 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2713fa3640 5 monclient: authenticate success, global_id 4310 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2713fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2708003610 con 0x7f270c0a6160 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 1 -- 172.21.15.121:0/70653269 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2708003b80 con 0x7f270c0a6160 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2710d3d640 10 monclient: dump: 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.198 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2713fa3640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.185+0000 7f2713fa3640 4 set_mon_vals no callback set 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7f2713fa3640 10 monclient: shutdown 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7f2713fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7f2713fa3640 1 -- 172.21.15.121:0/70653269 >> v1:172.21.15.121:6793/0 conn(0x7f270c0a6160 legacy=0x7f270c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7f2713fa3640 1 -- 172.21.15.121:0/70653269 shutdown_connections 2024-09-13T13:51:45.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7eff75d88640 1 -- 172.21.15.121:0/351390510 >> v1:172.21.15.121:6789/0 conn(0x7eff700a6160 legacy=0x7eff700a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7f2713fa3640 1 -- 172.21.15.121:0/70653269 >> 172.21.15.121:0/70653269 conn(0x7f270c0ab940 msgr2=0x7f270c0abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.186+0000 7eff76589640 1 -- 172.21.15.121:0/351390510 >> v1:172.21.15.121:6791/0 conn(0x7eff700a6c50 legacy=0x7eff70141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7f2713fa3640 1 -- 172.21.15.121:0/70653269 shutdown_connections 2024-09-13T13:51:45.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310651960 0 0) 0x7eff7009e210 con 0x7eff700a6160 2024-09-13T13:51:45.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7f2713fa3640 1 -- 172.21.15.121:0/70653269 wait complete. 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): my global_id is 4356 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff5c003520 con 0x7eff700a6160 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2446445872 0 0) 0x7eff700a7ef0 con 0x7eff700a6c50 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient(hunting): my global_id is 4270 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff7009e210 con 0x7eff700a6c50 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff75587640 1 -- 172.21.15.121:0/351390510 >> v1:172.21.15.121:6793/0 conn(0x7eff7009d150 legacy=0x7eff7013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933190238 0 0) 0x7eff7009e210 con 0x7eff700a6c50 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 >> v1:172.21.15.121:6793/0 conn(0x7eff7009d150 legacy=0x7eff7013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 >> v1:172.21.15.121:6789/0 conn(0x7eff700a6160 legacy=0x7eff700a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 monclient: found mon.noname-c 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7009e630 con 0x7eff700a6c50 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7eff68001e10 con 0x7eff700a6c50 2024-09-13T13:51:45.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7f2713fa3640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7f2713fa3640 10 monclient: monmap: 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.205 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.206 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff74d86640 10 monclient: dump: 2024-09-13T13:51:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.207 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.208 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff77fec640 5 monclient: authenticate success, global_id 4270 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.187+0000 7eff77fec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff68003990 con 0x7eff700a6c50 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff74d86640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff74d86640 1 -- 172.21.15.121:0/351390510 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7eff68003f00 con 0x7eff700a6c50 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff77fec640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff77fec640 4 set_mon_vals no callback set 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff74d86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff74d86640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff74d86640 10 monclient: dump: 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.209 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.210 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7f2713fa3640 1 Processor -- start 2024-09-13T13:51:45.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7f2713fa3640 1 -- start start 2024-09-13T13:51:45.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7f2713fa3640 10 monclient: init 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff77fec640 10 monclient: shutdown 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff77fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.188+0000 7eff77fec640 1 -- 172.21.15.121:0/351390510 >> v1:172.21.15.121:6791/0 conn(0x7eff700a6c50 legacy=0x7eff70141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7eff77fec640 1 -- 172.21.15.121:0/351390510 shutdown_connections 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7eff77fec640 1 -- 172.21.15.121:0/351390510 >> 172.21.15.121:0/351390510 conn(0x7eff700ab940 msgr2=0x7eff700abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7eff77fec640 1 -- 172.21.15.121:0/351390510 shutdown_connections 2024-09-13T13:51:45.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7eff77fec640 1 -- 172.21.15.121:0/351390510 wait complete. 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient: _add_conns ranks=[8,0,7,4,6,2,5,3,1] 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient(hunting): picked mon.noname-e con 0x7f270c09d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2711d3f640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 0x7f270c1397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56588/0 (socket says 172.21.15.121:56588) 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f270c0a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient(hunting): picked mon.noname-i con 0x7f270c0a6c50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c0a45a0 con 0x7f270c0a6160 2024-09-13T13:51:45.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c072bb0 con 0x7f270c0a6c50 2024-09-13T13:51:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2711d3f640 1 -- 172.21.15.121:0/566953217 learned_addr learned my addr 172.21.15.121:0/566953217 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c0716c0 con 0x7f270c09d150 2024-09-13T13:51:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.189+0000 7f2713fa3640 10 monclient(hunting): authenticate will time out at 784.819893s 2024-09-13T13:51:45.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.190+0000 7eff77fec640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.190+0000 7eff77fec640 10 monclient: monmap: 2024-09-13T13:51:45.215 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.216 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.190+0000 7eff77fec640 1 Processor -- start 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.190+0000 7eff77fec640 1 -- start start 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.190+0000 7eff77fec640 10 monclient: init 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient: _add_conns ranks=[8,7,6,0,3,2,4,5,1] 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient(hunting): picked mon.noname-e con 0x7eff7009d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient(hunting): picked mon.noname-i con 0x7eff700a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff76589640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7eff700a6c50 0x7eff701d61a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52314/0 (socket says 172.21.15.121:52314) 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff76589640 1 -- 172.21.15.121:0/2701558226 learned_addr learned my addr 172.21.15.121:0/2701558226 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient(hunting): picked mon.noname-d con 0x7eff700a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff70126ad0 con 0x7eff700a6c50 2024-09-13T13:51:45.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff700a4600 con 0x7eff700a6160 2024-09-13T13:51:45.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff700a8a80 con 0x7eff7009d150 2024-09-13T13:51:45.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7eff77fec640 10 monclient(hunting): authenticate will time out at 784.821465s 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7f271153e640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6789/0 conn(0x7f270c0a6160 legacy=0x7f270c1d6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.191+0000 7f2711d3f640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 legacy=0x7f270c1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363258203 0 0) 0x7f270c0716c0 con 0x7f270c09d150 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): my global_id is 4328 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f0003520 con 0x7f270c09d150 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148398571 0 0) 0x7f270c0a45a0 con 0x7f270c0a6160 2024-09-13T13:51:45.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): my global_id is 4365 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f270c0716c0 con 0x7f270c0a6160 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2712540640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.153:6792/0 conn(0x7f270c0a6c50 legacy=0x7f270c1da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2711d3f640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 legacy=0x7f270c1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2711d3f640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 legacy=0x7f270c1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2711d3f640 1 --1- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 0x7f270c1397c0 :-1 s=OPENED pgs=65 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2711d3f640 1 --1- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 0x7f270c1397c0 :-1 s=OPENED pgs=65 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6793/0 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3025760094 0 0) 0x7f270c0716c0 con 0x7f270c0a6160 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f270c0a45a0 con 0x7f270c0a6160 2024-09-13T13:51:45.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f26fc0047e0 con 0x7f270c0a6160 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): dump: 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.222 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1707527244 0 0) 0x7f270c0a45a0 con 0x7f270c0a6160 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6793/0 conn(0x7f270c09d150 legacy=0x7f270c1397c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.153:6792/0 conn(0x7f270c0a6c50 legacy=0x7f270c1da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 monclient: found mon.a 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c073a20 con 0x7f270c0a6160 2024-09-13T13:51:45.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 5 monclient: authenticate success, global_id 4365 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 10 monclient: _renew_subs 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f270c073180 con 0x7f270c0a6160 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f26fc004cd0 con 0x7f270c0a6160 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 10 monclient: _renew_subs 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2713fa3640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f270c072400 con 0x7f270c0a6160 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f26fc0051c0 con 0x7f270c0a6160 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:45.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f27027fc640 10 monclient: dump: 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.225 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.226 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.226 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.192+0000 7f2703fff640 4 set_mon_vals no callback set 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.193+0000 7eff76589640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.121:6792/0 conn(0x7eff700a6c50 legacy=0x7eff701d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.193+0000 7eff75d88640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.121:6793/0 conn(0x7eff7009d150 legacy=0x7eff701390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208334288 0 0) 0x7eff70126ad0 con 0x7eff700a6c50 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): my global_id is 4299 2024-09-13T13:51:45.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff54003520 con 0x7eff700a6c50 2024-09-13T13:51:45.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7f26effff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7f26effff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7f26effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f26fc005410 con 0x7f270c0a6160 2024-09-13T13:51:45.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2193459528 0 0) 0x7eff700a8a80 con 0x7eff7009d150 2024-09-13T13:51:45.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270c0710b0 con 0x7f270c0a6160 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): my global_id is 4337 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff70126ad0 con 0x7eff7009d150 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff75587640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.153:6792/0 conn(0x7eff700a6160 legacy=0x7eff701d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2159235280 0 0) 0x7eff54003520 con 0x7eff700a6c50 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7eff700a8a80 con 0x7eff700a6c50 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7eff68003e80 con 0x7eff700a6c50 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): dump: 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.231 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f26fc04e1f0 con 0x7f270c0a6160 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1737970755 0 0) 0x7eff700a8a80 con 0x7eff700a6c50 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.121:6793/0 conn(0x7eff7009d150 legacy=0x7eff701390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.153:6792/0 conn(0x7eff700a6160 legacy=0x7eff701d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 1 monclient: found mon.d 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.194+0000 7eff667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff667fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff70124770 con 0x7eff700a6c50 2024-09-13T13:51:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff667fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff667fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 5 monclient: authenticate success, global_id 4299 2024-09-13T13:51:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 10 monclient: _renew_subs 2024-09-13T13:51:45.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7f2711d3f640 10 monclient: get_auth_request con 0x7f26f003fc50 auth_method 0 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7eff70125450 con 0x7eff700a6c50 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 10 monclient: _renew_subs 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.195+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7eff70139800 con 0x7eff700a6c50 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.196+0000 7eff77fec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.196+0000 7eff77fec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.196+0000 7eff77fec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.196+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff700008d0 con 0x7eff700a6c50 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.196+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7eff68003990 con 0x7eff700a6c50 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7eff680046c0 con 0x7eff700a6c50 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 10 monclient: dump: 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.237 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7eff68012020 con 0x7eff700a6c50 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff67fff640 4 set_mon_vals no callback set 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.197+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7eff6804e4d0 con 0x7eff700a6c50 2024-09-13T13:51:45.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.198+0000 7eff75d88640 10 monclient: get_auth_request con 0x7eff5403fbc0 auth_method 0 2024-09-13T13:51:45.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.200+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f26fc017770 con 0x7f270c0a6160 2024-09-13T13:51:45.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.200+0000 7f27027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.200+0000 7f27027fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.202+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7eff68017b90 con 0x7eff700a6c50 2024-09-13T13:51:45.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.202+0000 7eff667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.202+0000 7eff667fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.227+0000 7f42591d5640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.227+0000 7f42591d5640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.228+0000 7f42591d5640 10 monclient: monmap: 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.229742+0000 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.229742+0000 2024-09-13T13:51:45.241 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.228+0000 7f42591d5640 1 Processor -- start 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.228+0000 7f42591d5640 1 -- start start 2024-09-13T13:51:45.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.228+0000 7f42591d5640 10 monclient: init 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient: _add_conns ranks=[7,3,4,2,0,1,5,8,6] 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient(hunting): picked mon.noname-i con 0x7f42540a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient(hunting): picked mon.noname-g con 0x7f425409d150 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient(hunting): picked mon.noname-c con 0x7f42540a6c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f425409e210 con 0x7f425409d150 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42540a7ef0 con 0x7f42540a6c50 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f425409d970 con 0x7f42540a6160 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f42591d5640 10 monclient(hunting): authenticate will time out at 784.859393s 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f425359e640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f42540a6c50 0x7f4254141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33770/0 (socket says 172.21.15.121:33770) 2024-09-13T13:51:45.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.229+0000 7f425359e640 1 -- 172.21.15.121:0/2421434823 learned_addr learned my addr 172.21.15.121:0/2421434823 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7f111e095640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7f111e095640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7f111e095640 10 monclient: monmap: 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.233175+0000 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.233175+0000 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.244 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7f425359e640 1 -- 172.21.15.121:0/2421434823 >> v1:172.21.15.121:6791/0 conn(0x7f42540a6c50 legacy=0x7f4254141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7fc9864e4640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7fc9864e4640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7fc9864e4640 10 monclient: monmap: 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.233383+0000 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.233383+0000 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.246 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.247 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7f425259c640 1 -- 172.21.15.121:0/2421434823 >> v1:172.21.15.153:6790/0 conn(0x7f425409d150 legacy=0x7f425413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.231+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718911504 0 0) 0x7f42540a7ef0 con 0x7f42540a6c50 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): my global_id is 4279 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4252d9d640 1 -- 172.21.15.121:0/2421434823 >> v1:172.21.15.153:6792/0 conn(0x7f42540a6160 legacy=0x7f42540a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f423c003520 con 0x7f42540a6c50 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3973363041 0 0) 0x7f425409e210 con 0x7f425409d150 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): my global_id is 4269 2024-09-13T13:51:45.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42540a7ef0 con 0x7f425409d150 2024-09-13T13:51:45.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f111e095640 1 Processor -- start 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 560379096 0 0) 0x7f425409d970 con 0x7f42540a6160 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient(hunting): my global_id is 4354 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f425409e210 con 0x7f42540a6160 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3347801943 0 0) 0x7f42540a7ef0 con 0x7f425409d150 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 >> v1:172.21.15.153:6792/0 conn(0x7f42540a6160 legacy=0x7f42540a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 >> v1:172.21.15.121:6791/0 conn(0x7f42540a6c50 legacy=0x7f4254141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 monclient: found mon.noname-g 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f425409e630 con 0x7f425409d150 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4248001b10 con 0x7f425409d150 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f111e095640 1 -- start start 2024-09-13T13:51:45.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: dump: 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.252 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.253 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.253 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.253 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f111e095640 10 monclient: init 2024-09-13T13:51:45.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f42591d5640 5 monclient: authenticate success, global_id 4269 2024-09-13T13:51:45.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f42591d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4248003610 con 0x7f425409d150 2024-09-13T13:51:45.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 1 -- 172.21.15.121:0/2421434823 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4248003b40 con 0x7f425409d150 2024-09-13T13:51:45.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7fc9864e4640 1 Processor -- start 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f4251d9b640 10 monclient: dump: 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.255 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f42591d5640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7f42591d5640 4 set_mon_vals no callback set 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7fc9864e4640 1 -- start start 2024-09-13T13:51:45.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.232+0000 7fc9864e4640 10 monclient: init 2024-09-13T13:51:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient: _add_conns ranks=[8,3,0,7,6,4,5,1,2] 2024-09-13T13:51:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient(hunting): picked mon.noname-e con 0x7f11180a1350 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient(hunting): picked mon.noname-g con 0x7f11180a1e80 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:45.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient(hunting): picked mon.noname-a con 0x7f1118143a60 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111809e410 con 0x7f1118143a60 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111809e6f0 con 0x7f11180a1e80 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111809e9d0 con 0x7f11180a1350 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f111e095640 10 monclient(hunting): authenticate will time out at 784.863393s 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f1c15f56640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f1c15f56640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f1c15f56640 10 monclient: monmap: 2024-09-13T13:51:45.258 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.235086+0000 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.235086+0000 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.259 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f1117fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1118143a60 0x7f1118145e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45496/0 (socket says 172.21.15.121:45496) 2024-09-13T13:51:45.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f1117fff640 1 -- 172.21.15.121:0/4188531895 learned_addr learned my addr 172.21.15.121:0/4188531895 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f42591d5640 10 monclient: shutdown 2024-09-13T13:51:45.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f42591d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f42591d5640 1 -- 172.21.15.121:0/2421434823 >> v1:172.21.15.153:6790/0 conn(0x7f425409d150 legacy=0x7f425413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7fc9864e4640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f42591d5640 1 -- 172.21.15.121:0/2421434823 shutdown_connections 2024-09-13T13:51:45.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7f42591d5640 1 -- 172.21.15.121:0/2421434823 >> 172.21.15.121:0/2421434823 conn(0x7f42540ab940 msgr2=0x7f42540abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7fc9864e4640 10 monclient: _add_conns ranks=[7,6,0,4,5,2,3,8,1] 2024-09-13T13:51:45.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.233+0000 7fc9864e4640 10 monclient(hunting): picked mon.noname-i con 0x7fc9800a3e30 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:45.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 10 monclient(hunting): picked mon.noname-d con 0x7fc9800a4210 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f42591d5640 1 -- 172.21.15.121:0/2421434823 shutdown_connections 2024-09-13T13:51:45.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc98013d050 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9800737f0 con 0x7fc98013d050 2024-09-13T13:51:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f42591d5640 1 -- 172.21.15.121:0/2421434823 wait complete. 2024-09-13T13:51:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc97f7fe640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fc9800a4210 0x7fc9800a4640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52330/0 (socket says 172.21.15.121:52330) 2024-09-13T13:51:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc97f7fe640 1 -- 172.21.15.121:0/1189785378 learned_addr learned my addr 172.21.15.121:0/1189785378 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9800734a0 con 0x7fc9800a4210 2024-09-13T13:51:45.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9800a5b60 con 0x7fc9800a3e30 2024-09-13T13:51:45.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7fc9864e4640 10 monclient(hunting): authenticate will time out at 784.864357s 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 1 Processor -- start 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 1 -- start start 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 10 monclient: init 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 10 monclient: _add_conns ranks=[0,3,4,7,8,6,1,5,2] 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 10 monclient(hunting): picked mon.noname-a con 0x7f1c100a1e10 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 10 monclient(hunting): picked mon.noname-g con 0x7f1c1009ffb0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.234+0000 7f1c15f56640 10 monclient(hunting): picked mon.noname-c con 0x7f1c101330e0 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c15f56640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c100a1050 con 0x7f1c100a1e10 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c15f56640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c100a1330 con 0x7f1c1009ffb0 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c0f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1c100a1e10 0x7f1c101329d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45510/0 (socket says 172.21.15.121:45510) 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c0f7fe640 1 -- 172.21.15.121:0/4274472944 learned_addr learned my addr 172.21.15.121:0/4274472944 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c0effd640 1 --1- 172.21.15.121:0/4274472944 >> v1:172.21.15.153:6790/0 conn(0x7f1c1009ffb0 0x7f1c1013ace0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:47952/0 (socket says 172.21.15.121:47952) 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c15f56640 1 -- 172.21.15.121:0/4274472944 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c100a1610 con 0x7f1c101330e0 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c15f56640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1c15f56640 10 monclient(hunting): authenticate will time out at 784.865179s 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f42591d5640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f42591d5640 10 monclient: monmap: 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.266 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.267 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1117fff640 1 -- 172.21.15.121:0/4188531895 >> v1:172.21.15.121:6789/0 conn(0x7f1118143a60 legacy=0x7f1118145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f1116ffd640 1 -- 172.21.15.121:0/4188531895 >> v1:172.21.15.153:6790/0 conn(0x7f11180a1e80 legacy=0x7f1118142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11177fe640 1 -- 172.21.15.121:0/4188531895 >> v1:172.21.15.121:6793/0 conn(0x7f11180a1350 legacy=0x7f11180a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103478180 0 0) 0x7f111809e410 con 0x7f1118143a60 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11167fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11167fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11167fc640 10 monclient(hunting): my global_id is 4374 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.235+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1100003520 con 0x7f1118143a60 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 137515270 0 0) 0x7f111809e6f0 con 0x7f11180a1e80 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): my global_id is 4278 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f111809e410 con 0x7f11180a1e80 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2891382490 0 0) 0x7f111809e9d0 con 0x7f11180a1350 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient(hunting): my global_id is 4346 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f111809e6f0 con 0x7f11180a1350 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2598215631 0 0) 0x7f111809e410 con 0x7f11180a1e80 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 >> v1:172.21.15.121:6793/0 conn(0x7f11180a1350 legacy=0x7f11180a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 >> v1:172.21.15.121:6789/0 conn(0x7f1118143a60 legacy=0x7f1118145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 monclient: found mon.noname-g 2024-09-13T13:51:45.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f42591d5640 1 Processor -- start 2024-09-13T13:51:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1118068be0 con 0x7f11180a1e80 2024-09-13T13:51:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f110c001b10 con 0x7f11180a1e80 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: dump: 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.272 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f111e095640 5 monclient: authenticate success, global_id 4278 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f111e095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f42591d5640 1 -- start start 2024-09-13T13:51:45.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f42591d5640 10 monclient: init 2024-09-13T13:51:45.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97f7fe640 1 -- 172.21.15.121:0/1189785378 >> v1:172.21.15.121:6792/0 conn(0x7fc9800a4210 legacy=0x7fc9800a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc984a81640 1 -- 172.21.15.121:0/1189785378 >> v1:172.21.15.121:6789/0 conn(0x7fc98013d050 legacy=0x7fc98013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f110c003610 con 0x7f11180a1e80 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 1 -- 172.21.15.121:0/4188531895 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f110c003bc0 con 0x7f11180a1e80 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f11167fc640 10 monclient: dump: 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.275 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f111e095640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7f111e095640 4 set_mon_vals no callback set 2024-09-13T13:51:45.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1948775003 0 0) 0x7fc9800737f0 con 0x7fc98013d050 2024-09-13T13:51:45.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 10 monclient(hunting): my global_id is 4383 2024-09-13T13:51:45.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc964003520 con 0x7fc98013d050 2024-09-13T13:51:45.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 753931127 0 0) 0x7fc964003520 con 0x7fc98013d050 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 >> v1:172.21.15.153:6792/0 conn(0x7fc9800a3e30 legacy=0x7fc980073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.236+0000 7fc97ffff640 1 -- 172.21.15.121:0/1189785378 >> v1:172.21.15.153:6792/0 conn(0x7fc9800a3e30 legacy=0x7fc980073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 >> v1:172.21.15.121:6792/0 conn(0x7fc9800a4210 legacy=0x7fc9800a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 1 monclient: found mon.noname-a 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc980143b90 con 0x7fc98013d050 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc970001b10 con 0x7fc98013d050 2024-09-13T13:51:45.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: dump: 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.282 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc9864e4640 5 monclient: authenticate success, global_id 4383 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc9864e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 10 monclient: shutdown 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 1 -- 172.21.15.121:0/4188531895 >> v1:172.21.15.153:6790/0 conn(0x7f11180a1e80 legacy=0x7f1118142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0ffff640 1 -- 172.21.15.121:0/4274472944 >> v1:172.21.15.121:6791/0 conn(0x7f1c101330e0 legacy=0x7f1c1013e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0effd640 1 -- 172.21.15.121:0/4274472944 >> v1:172.21.15.153:6790/0 conn(0x7f1c1009ffb0 legacy=0x7f1c1013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 1 -- 172.21.15.121:0/4188531895 shutdown_connections 2024-09-13T13:51:45.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f42591d5640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 1 -- 172.21.15.121:0/4188531895 >> 172.21.15.121:0/4188531895 conn(0x7f11180a48d0 msgr2=0x7f11180a4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f42591d5640 10 monclient: _add_conns ranks=[4,8,6,2,3,1,7,0,5] 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0f7fe640 1 -- 172.21.15.121:0/4274472944 >> v1:172.21.15.121:6789/0 conn(0x7f1c100a1e10 legacy=0x7f1c101329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211403234 0 0) 0x7f1c100a1330 con 0x7f1c1009ffb0 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): my global_id is 4287 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf8003520 con 0x7f1c1009ffb0 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc970003610 con 0x7fc98013d050 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 1 -- 172.21.15.121:0/1189785378 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc970003b80 con 0x7fc98013d050 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc97effd640 10 monclient: dump: 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.286 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.287 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 1 -- 172.21.15.121:0/4188531895 shutdown_connections 2024-09-13T13:51:45.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1419838102 0 0) 0x7f1c100a1050 con 0x7f1c100a1e10 2024-09-13T13:51:45.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f111e095640 1 -- 172.21.15.121:0/4188531895 wait complete. 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc9864e4640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): my global_id is 4392 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c100a1330 con 0x7f1c100a1e10 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523711819 0 0) 0x7f1c100a1610 con 0x7f1c101330e0 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 10 monclient(hunting): my global_id is 4288 2024-09-13T13:51:45.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7fc9864e4640 4 set_mon_vals no callback set 2024-09-13T13:51:45.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bf8003a10 con 0x7f1c101330e0 2024-09-13T13:51:45.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f42591d5640 10 monclient(hunting): picked mon.noname-c con 0x7f425409d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f4252d9d640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f425409d150 0x7f42541391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33788/0 (socket says 172.21.15.121:33788) 2024-09-13T13:51:45.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f42591d5640 10 monclient(hunting): picked mon.noname-e con 0x7f42540a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.237+0000 7f4252d9d640 1 -- 172.21.15.121:0/2353594004 learned_addr learned my addr 172.21.15.121:0/2353594004 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4034490024 0 0) 0x7f1bf8003520 con 0x7f1c1009ffb0 2024-09-13T13:51:45.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 >> v1:172.21.15.121:6791/0 conn(0x7f1c101330e0 legacy=0x7f1c1013e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 >> v1:172.21.15.121:6789/0 conn(0x7f1c100a1e10 legacy=0x7f1c101329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 monclient: found mon.noname-g 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c100a2560 con 0x7f1c1009ffb0 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1c04001a10 con 0x7f1c1009ffb0 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: dump: 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.292 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c15f56640 5 monclient: authenticate success, global_id 4287 2024-09-13T13:51:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c15f56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f42591d5640 10 monclient(hunting): picked mon.noname-d con 0x7f42540a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4254126890 con 0x7f425409d150 2024-09-13T13:51:45.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f111e095640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c04001ce0 con 0x7f1c1009ffb0 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 1 -- 172.21.15.121:0/4274472944 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1c04004b20 con 0x7f1c1009ffb0 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c0e7fc640 10 monclient: dump: 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.295 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c15f56640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f1c15f56640 4 set_mon_vals no callback set 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42540a4060 con 0x7f42540a6c50 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f111e095640 10 monclient: monmap: 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.297 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42541265b0 con 0x7f42540a6160 2024-09-13T13:51:45.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f42591d5640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f42591d5640 10 monclient(hunting): authenticate will time out at 784.868571s 2024-09-13T13:51:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 10 monclient: shutdown 2024-09-13T13:51:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 >> v1:172.21.15.121:6789/0 conn(0x7fc98013d050 legacy=0x7fc98013f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 shutdown_connections 2024-09-13T13:51:45.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 >> 172.21.15.121:0/1189785378 conn(0x7fc9800a9430 msgr2=0x7fc9800a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 shutdown_connections 2024-09-13T13:51:45.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7f111e095640 1 Processor -- start 2024-09-13T13:51:45.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.238+0000 7fc9864e4640 1 -- 172.21.15.121:0/1189785378 wait complete. 2024-09-13T13:51:45.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f111e095640 1 -- start start 2024-09-13T13:51:45.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f111e095640 10 monclient: init 2024-09-13T13:51:45.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 10 monclient: shutdown 2024-09-13T13:51:45.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 1 -- 172.21.15.121:0/4274472944 >> v1:172.21.15.153:6790/0 conn(0x7f1c1009ffb0 legacy=0x7f1c1013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 1 -- 172.21.15.121:0/4274472944 shutdown_connections 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 1 -- 172.21.15.121:0/4274472944 >> 172.21.15.121:0/4274472944 conn(0x7f1c100a8340 msgr2=0x7f1c100a8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 1 -- 172.21.15.121:0/4274472944 shutdown_connections 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f1c15f56640 1 -- 172.21.15.121:0/4274472944 wait complete. 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7fc9864e4640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7fc9864e4640 10 monclient: monmap: 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.302 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.303 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f111e095640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.239+0000 7f111e095640 10 monclient: _add_conns ranks=[6,2,8,1,3,7,5,4,0] 2024-09-13T13:51:45.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f425259c640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.121:6793/0 conn(0x7f42540a6160 legacy=0x7f42541d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f4252d9d640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.121:6791/0 conn(0x7f425409d150 legacy=0x7f42541391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 10 monclient(hunting): picked mon.noname-d con 0x7f11180a1350 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 10 monclient(hunting): picked mon.noname-b con 0x7f11180a1e80 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:45.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 10 monclient(hunting): picked mon.noname-e con 0x7f1118143a60 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:45.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11181408b0 con 0x7f11180a1e80 2024-09-13T13:51:45.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853359579 0 0) 0x7f4254126890 con 0x7f425409d150 2024-09-13T13:51:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): my global_id is 4297 2024-09-13T13:51:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4240003520 con 0x7f425409d150 2024-09-13T13:51:45.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f1117fff640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f1118143a60 0x7f11181da5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56628/0 (socket says 172.21.15.121:56628) 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1118129150 con 0x7f11180a1350 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f1117fff640 1 -- 172.21.15.121:0/1624379884 learned_addr learned my addr 172.21.15.121:0/1624379884 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111812b4b0 con 0x7f1118143a60 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f11177fe640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f11180a1350 0x7f111813fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52356/0 (socket says 172.21.15.121:52356) 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f111e095640 10 monclient(hunting): authenticate will time out at 784.870442s 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4216847140 0 0) 0x7f42541265b0 con 0x7f42540a6160 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): my global_id is 4355 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4254126890 con 0x7f42540a6160 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f425359e640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.121:6792/0 conn(0x7f42540a6c50 legacy=0x7f42541d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f1c15f56640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f1c15f56640 10 monclient: monmap: 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.308 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.309 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3448774029 0 0) 0x7f4240003520 con 0x7f425409d150 2024-09-13T13:51:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42541265b0 con 0x7f425409d150 2024-09-13T13:51:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4244003610 con 0x7f425409d150 2024-09-13T13:51:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7fc9864e4640 1 Processor -- start 2024-09-13T13:51:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.311 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.312 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181310701 0 0) 0x7f42540a4060 con 0x7f42540a6c50 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): my global_id is 4308 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4240003520 con 0x7f42540a6c50 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3769914465 0 0) 0x7f4254126890 con 0x7f42540a6160 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f42540a4060 con 0x7f42540a6160 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4248003b00 con 0x7f42540a6160 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.240+0000 7f423b7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.314 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1873007001 0 0) 0x7f42541265b0 con 0x7f425409d150 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 1 -- start start 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.121:6793/0 conn(0x7f42540a6160 legacy=0x7f42541d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.121:6792/0 conn(0x7f42540a6c50 legacy=0x7f42541d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 monclient: found mon.c 2024-09-13T13:51:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42541398c0 con 0x7f425409d150 2024-09-13T13:51:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 10 monclient: init 2024-09-13T13:51:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4244003c00 con 0x7f425409d150 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f4244004170 con 0x7f425409d150 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:45.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f1c15f56640 1 Processor -- start 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f423b7fe640 10 monclient: dump: 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.318 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f42591d5640 5 monclient: authenticate success, global_id 4297 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f42591d5640 10 monclient: _renew_subs 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f42591d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4254139b50 con 0x7f425409d150 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f425159a640 4 set_mon_vals no callback set 2024-09-13T13:51:45.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 10 monclient: _add_conns ranks=[4,2,0,6,7,5,1,3,8] 2024-09-13T13:51:45.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f1c15f56640 1 -- start start 2024-09-13T13:51:45.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7f1c15f56640 10 monclient: init 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc9800a3e30 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc9800a4210 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc98013d050 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc98013c720 con 0x7fc98013d050 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc980143380 con 0x7fc9800a4210 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.241+0000 7fc9864e4640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc980140780 con 0x7fc9800a3e30 2024-09-13T13:51:45.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7fc9864e4640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7fc97f7fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fc9800a4210 0x7fc9801d2de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59152/0 (socket says 172.21.15.121:59152) 2024-09-13T13:51:45.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7fc9864e4640 10 monclient(hunting): authenticate will time out at 784.872026s 2024-09-13T13:51:45.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7fc97f7fe640 1 -- 172.21.15.121:0/2953689295 learned_addr learned my addr 172.21.15.121:0/2953689295 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f42591d5640 10 monclient: _renew_subs 2024-09-13T13:51:45.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f42591d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f42541cd7f0 con 0x7f425409d150 2024-09-13T13:51:45.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f1117fff640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.121:6793/0 conn(0x7f1118143a60 legacy=0x7f11181da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f1116ffd640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.121:6790/0 conn(0x7f11180a1e80 legacy=0x7f11181d6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f11177fe640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.121:6792/0 conn(0x7f11180a1350 legacy=0x7f111813fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f4244004310 con 0x7f425409d150 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.242+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024621560 0 0) 0x7f11181408b0 con 0x7f11180a1e80 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): my global_id is 4277 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1104003520 con 0x7f11180a1e80 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432594376 0 0) 0x7f111812b4b0 con 0x7f1118143a60 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): my global_id is 4364 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11181408b0 con 0x7f1118143a60 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632427816 0 0) 0x7f1118129150 con 0x7f11180a1350 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): my global_id is 4317 2024-09-13T13:51:45.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f111812b4b0 con 0x7f11180a1350 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f42591d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f42591d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f42591d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4254124810 con 0x7f425409d150 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f424404e4e0 con 0x7f425409d150 2024-09-13T13:51:45.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2542725534 0 0) 0x7f1104003520 con 0x7f11180a1e80 2024-09-13T13:51:45.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f1c15f56640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f1c15f56640 10 monclient: _add_conns ranks=[1,5,2,7,3,6,4,0,8] 2024-09-13T13:51:45.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f1c15f56640 10 monclient(hunting): picked mon.noname-f con 0x7f1c1009ffb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:45.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1118129150 con 0x7f11180a1e80 2024-09-13T13:51:45.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 650311841 0 0) 0x7f11181408b0 con 0x7f1118143a60 2024-09-13T13:51:45.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f1c15f56640 10 monclient(hunting): picked mon.noname-h con 0x7f1c100a1e10 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:45.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1104003520 con 0x7f1118143a60 2024-09-13T13:51:45.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f110c003b00 con 0x7f11180a1e80 2024-09-13T13:51:45.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): dump: 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.329 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1110002060 con 0x7f1118143a60 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): dump: 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.330 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1310539303 0 0) 0x7f111812b4b0 con 0x7f11180a1350 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11181408b0 con 0x7f11180a1350 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1108003610 con 0x7f11180a1350 2024-09-13T13:51:45.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f10fffff640 10 monclient(hunting): dump: 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.332 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.333 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.333 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.333 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.333 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f425259c640 10 monclient: get_auth_request con 0x7f42400401c0 auth_method 0 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f1c0f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1c1009ffb0 0x7f1c101b84d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.121:39072/0 (socket says 172.21.15.121:39072) 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.243+0000 7f1c0effd640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f1c100a1e10 0x7f1c101b8be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:45094/0 (socket says 172.21.15.121:45094) 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c15f56640 10 monclient(hunting): picked mon.noname-b con 0x7f1c101330e0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c0f7fe640 1 -- 172.21.15.121:0/43442838 learned_addr learned my addr 172.21.15.121:0/43442838 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c15f56640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c101b92f0 con 0x7f1c1009ffb0 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c15f56640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c101b9600 con 0x7f1c101330e0 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c15f56640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c101bc1e0 con 0x7f1c100a1e10 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4112186135 0 0) 0x7f1118129150 con 0x7f11180a1e80 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.121:6793/0 conn(0x7f1118143a60 legacy=0x7f11181da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.121:6792/0 conn(0x7f11180a1350 legacy=0x7f111813fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 1 monclient: found mon.b 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1118129cb0 con 0x7f11180a1e80 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f10fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 5 monclient: authenticate success, global_id 4277 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 10 monclient: _renew_subs 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1118140220 con 0x7f11180a1e80 2024-09-13T13:51:45.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c15f56640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f1c15f56640 10 monclient(hunting): authenticate will time out at 784.874429s 2024-09-13T13:51:45.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97f7fe640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.121:6790/0 conn(0x7fc9800a4210 legacy=0x7fc9801d2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97ffff640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.121:6791/0 conn(0x7fc9800a3e30 legacy=0x7fc98013c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 10 monclient: _renew_subs 2024-09-13T13:51:45.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f11181289e0 con 0x7f11180a1e80 2024-09-13T13:51:45.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc984a81640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.121:6789/0 conn(0x7fc98013d050 legacy=0x7fc9801d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2005958641 0 0) 0x7fc9800734a0 con 0x7fc9800a4210 2024-09-13T13:51:45.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): my global_id is 4286 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc968003550 con 0x7fc9800a4210 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486776087 0 0) 0x7fc98013c720 con 0x7fc98013d050 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 10 monclient(hunting): my global_id is 4401 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.244+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9800734a0 con 0x7fc98013d050 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3402752069 0 0) 0x7fc968003550 con 0x7fc9800a4210 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc98013c720 con 0x7fc9800a4210 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc96c001d00 con 0x7fc9800a4210 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): dump: 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.341 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 135696300 0 0) 0x7fc9800734a0 con 0x7fc98013d050 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc968003550 con 0x7fc98013d050 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc970003b00 con 0x7fc98013d050 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1c0effd640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.153:6791/0 conn(0x7f1c100a1e10 legacy=0x7f1c101b8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient(hunting): dump: 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.343 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3452815490 0 0) 0x7fc98013c720 con 0x7fc9800a4210 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.121:6791/0 conn(0x7fc9800a3e30 legacy=0x7fc98013c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.121:6789/0 conn(0x7fc98013d050 legacy=0x7fc9801d6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 monclient: found mon.b 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98006f0c0 con 0x7fc9800a4210 2024-09-13T13:51:45.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 5 monclient: authenticate success, global_id 4286 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 10 monclient: _renew_subs 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc98006f3a0 con 0x7fc9800a4210 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 10 monclient: _renew_subs 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc9801c8a70 con 0x7fc9800a4210 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc96c004ce0 con 0x7fc9800a4210 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc96c0051d0 con 0x7fc9800a4210 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97e7fc640 4 set_mon_vals no callback set 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.245+0000 7fc97cff9640 10 monclient: dump: 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.346 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720747220 0 0) 0x7f1c101bc1e0 con 0x7f1c100a1e10 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): my global_id is 4298 2024-09-13T13:51:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bfc003520 con 0x7f1c100a1e10 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f111e095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f111e095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f111e095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111812ade0 con 0x7f11180a1e80 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1c0f7fe640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.153:6789/0 conn(0x7f1c1009ffb0 legacy=0x7f1c101b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2130526026 0 0) 0x7f1c101b92f0 con 0x7f1c1009ffb0 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 10 monclient(hunting): my global_id is 4321 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c101bc1e0 con 0x7f1c1009ffb0 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.246+0000 7fc9864e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7fc9864e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7fc9864e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1650041932 0 0) 0x7f1bfc003520 con 0x7f1c100a1e10 2024-09-13T13:51:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c101b92f0 con 0x7f1c100a1e10 2024-09-13T13:51:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1c04004b20 con 0x7f1c100a1e10 2024-09-13T13:51:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:51:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95c005080 con 0x7fc9800a4210 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): dump: 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.351 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1395924927 0 0) 0x7f1c101bc1e0 con 0x7f1c1009ffb0 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1bfc003520 con 0x7f1c1009ffb0 2024-09-13T13:51:45.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1c000047e0 con 0x7f1c1009ffb0 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient(hunting): dump: 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.353 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c0ffff640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.121:6790/0 conn(0x7f1c101330e0 legacy=0x7f1c101d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1773698614 0 0) 0x7f1c101b92f0 con 0x7f1c100a1e10 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.121:6790/0 conn(0x7f1c101330e0 legacy=0x7f1c101d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.153:6789/0 conn(0x7f1c1009ffb0 legacy=0x7f1c101b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 monclient: found mon.h 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c101ba560 con 0x7f1c100a1e10 2024-09-13T13:51:45.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1bf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 5 monclient: authenticate success, global_id 4298 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 10 monclient: _renew_subs 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c101ba840 con 0x7f1c100a1e10 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 10 monclient: _renew_subs 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.247+0000 7f1c15f56640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c101bb830 con 0x7f1c100a1e10 2024-09-13T13:51:45.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f110c003610 con 0x7f11180a1e80 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f110c004470 con 0x7f11180a1e80 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 10 monclient: dump: 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.356 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f110c011ca0 con 0x7f11180a1e80 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f110c04e1b0 con 0x7f11180a1e80 2024-09-13T13:51:45.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.248+0000 7f1115ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf5ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf0005080 con 0x7f1c100a1e10 2024-09-13T13:51:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fc96c005370 con 0x7fc9800a4210 2024-09-13T13:51:45.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f11177fe640 10 monclient: get_auth_request con 0x7f11040404e0 auth_method 0 2024-09-13T13:51:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7fc96c04e250 con 0x7fc9800a4210 2024-09-13T13:51:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7fc97ffff640 10 monclient: get_auth_request con 0x7fc968040110 auth_method 0 2024-09-13T13:51:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f4254124810 con 0x7f425409d150 2024-09-13T13:51:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f423b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f423b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c04001ce0 con 0x7f1c100a1e10 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf7fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1c040014b0 con 0x7f1c100a1e10 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.249+0000 7f1bf7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.250+0000 7f1bf7fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.250+0000 7f1c0d7fa640 4 set_mon_vals no callback set 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.250+0000 7f1bf7fff640 10 monclient: dump: 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.361 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.250+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1c04011970 con 0x7f1c100a1e10 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.250+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f1c0404d3b0 con 0x7f1c100a1e10 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.252+0000 7f1c0f7fe640 10 monclient: get_auth_request con 0x7f1bfc040160 auth_method 0 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.254+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f1c04017880 con 0x7f1c100a1e10 2024-09-13T13:51:45.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.254+0000 7f1bf7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.254+0000 7f1bf7fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.257+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fc980143380 con 0x7fc9800a4210 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.257+0000 7fc97cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.257+0000 7fc97cff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.266+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f110c017850 con 0x7f11180a1e80 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.266+0000 7f10fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.266+0000 7f10fffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.422+0000 7f26effff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.422+0000 7f26effff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.422+0000 7f26effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.423+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f270c0008d0 con 0x7f270c0a6160 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.423+0000 7f27027fc640 1 -- 172.21.15.121:0/566953217 <== mon.0 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 532354816) 0x7f26fc017070 con 0x7f270c0a6160 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.423+0000 7f27027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.423+0000 7f27027fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:45.425 INFO:teuthology.orchestra.run.smithi121.stdout:51539607557 2024-09-13T13:51:45.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.153:6812/1206688214 conn(0x7f26f003fc50 legacy=0x7f26f0042110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 10 monclient: shutdown 2024-09-13T13:51:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 >> v1:172.21.15.121:6789/0 conn(0x7f270c0a6160 legacy=0x7f270c1d6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 shutdown_connections 2024-09-13T13:51:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 >> 172.21.15.121:0/566953217 conn(0x7f270c0ab940 msgr2=0x7f270c1c1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 shutdown_connections 2024-09-13T13:51:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.425+0000 7f26effff640 1 -- 172.21.15.121:0/566953217 wait complete. 2024-09-13T13:51:45.439 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.0 2024-09-13T13:51:45.439 DEBUG:teuthology.parallel:result is None 2024-09-13T13:51:45.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf5ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T13:51:45.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf5ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T13:51:45.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf5ffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:51:45.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1bf0005370 con 0x7f1c100a1e10 2024-09-13T13:51:45.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf7fff640 1 -- 172.21.15.121:0/43442838 <== mon.5 v1:172.21.15.153:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 2777258794) 0x7f1c04017180 con 0x7f1c100a1e10 2024-09-13T13:51:45.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-13T13:51:45.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.454+0000 7f1bf7fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:45.457 INFO:teuthology.orchestra.run.smithi121.stdout:55834574853 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1bfc040160 legacy=0x7f1bfc042620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 10 monclient: shutdown 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 >> v1:172.21.15.153:6791/0 conn(0x7f1c100a1e10 legacy=0x7f1c101b8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 shutdown_connections 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 >> 172.21.15.121:0/43442838 conn(0x7f1c100a8340 msgr2=0x7f1c1009a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 shutdown_connections 2024-09-13T13:51:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.457+0000 7f1bf5ffb640 1 -- 172.21.15.121:0/43442838 wait complete. 2024-09-13T13:51:45.469 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-09-13T13:51:45.469 DEBUG:teuthology.parallel:result is None 2024-09-13T13:51:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff77fec640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T13:51:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff77fec640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T13:51:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff77fec640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7eff701cd710 con 0x7eff700a6c50 2024-09-13T13:51:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff667fc640 1 -- 172.21.15.121:0/2701558226 <== mon.6 v1:172.21.15.121:6792/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 1902013079) 0x7eff68020690 con 0x7eff700a6c50 2024-09-13T13:51:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-13T13:51:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.556+0000 7eff667fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:45.559 INFO:teuthology.orchestra.run.smithi121.stdout:47244640262 2024-09-13T13:51:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.153:6812/1206688214 conn(0x7eff5403fbc0 legacy=0x7eff54042080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 10 monclient: shutdown 2024-09-13T13:51:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 >> v1:172.21.15.121:6792/0 conn(0x7eff700a6c50 legacy=0x7eff701d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 shutdown_connections 2024-09-13T13:51:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 >> 172.21.15.121:0/2701558226 conn(0x7eff700ab940 msgr2=0x7eff700aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 shutdown_connections 2024-09-13T13:51:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.558+0000 7eff77fec640 1 -- 172.21.15.121:0/2701558226 wait complete. 2024-09-13T13:51:45.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.2 2024-09-13T13:51:45.571 DEBUG:teuthology.parallel:result is None 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f42591d5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f42591d5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f42591d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4254074b10 con 0x7f425409d150 2024-09-13T13:51:45.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f423b7fe640 1 -- 172.21.15.121:0/2353594004 <== mon.4 v1:172.21.15.121:6791/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 203001719) 0x7f42440178a0 con 0x7f425409d150 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f423b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.591+0000 7f423b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:45.594 INFO:teuthology.orchestra.run.smithi121.stdout:51539607556 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.153:6812/1206688214 conn(0x7f42400401c0 legacy=0x7f4240042680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 10 monclient: shutdown 2024-09-13T13:51:45.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 >> v1:172.21.15.121:6791/0 conn(0x7f425409d150 legacy=0x7f42541391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 shutdown_connections 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 >> 172.21.15.121:0/2353594004 conn(0x7f42540ab940 msgr2=0x7f42541c1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 shutdown_connections 2024-09-13T13:51:45.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.593+0000 7f42591d5640 1 -- 172.21.15.121:0/2353594004 wait complete. 2024-09-13T13:51:45.606 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.3 2024-09-13T13:51:45.606 DEBUG:teuthology.parallel:result is None 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc9864e4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc9864e4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc9864e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc95c005370 con 0x7fc9800a4210 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc97cff9640 1 -- 172.21.15.121:0/2953689295 <== mon.2 v1:172.21.15.121:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 2194451396) 0x7fc96c0178b0 con 0x7fc9800a4210 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc97cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-13T13:51:45.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.637+0000 7fc97cff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:45.640 INFO:teuthology.orchestra.run.smithi121.stdout:55834574851 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.153:6812/1206688214 conn(0x7fc968040110 legacy=0x7fc9680425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 10 monclient: shutdown 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 >> v1:172.21.15.121:6790/0 conn(0x7fc9800a4210 legacy=0x7fc9801d2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 shutdown_connections 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 >> 172.21.15.121:0/2953689295 conn(0x7fc9800a9430 msgr2=0x7fc9800a7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 shutdown_connections 2024-09-13T13:51:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.639+0000 7fc9864e4640 1 -- 172.21.15.121:0/2953689295 wait complete. 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f111e095640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f111e095640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f111e095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:45.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f111806de90 con 0x7f11180a1e80 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f10fffff640 1 -- 172.21.15.121:0/1624379884 <== mon.2 v1:172.21.15.121:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 3887400221) 0x7f110c017150 con 0x7f11180a1e80 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f10fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.645+0000 7f10fffff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:45.648 INFO:teuthology.orchestra.run.smithi121.stdout:60129542148 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.153:6812/1206688214 conn(0x7f11040404e0 legacy=0x7f11040429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 10 monclient: shutdown 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 >> v1:172.21.15.121:6790/0 conn(0x7f11180a1e80 legacy=0x7f11181d6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 shutdown_connections 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 >> 172.21.15.121:0/1624379884 conn(0x7f11180a48d0 msgr2=0x7f11180a0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 shutdown_connections 2024-09-13T13:51:45.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.648+0000 7f111e095640 1 -- 172.21.15.121:0/1624379884 wait complete. 2024-09-13T13:51:45.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2024-09-13T13:51:45.652 DEBUG:teuthology.parallel:result is None 2024-09-13T13:51:45.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.1 2024-09-13T13:51:45.660 DEBUG:teuthology.parallel:result is None 2024-09-13T13:51:45.660 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:51:45.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:45.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.797+0000 7fd41fc60640 10 monclient: get_monmap_and_config 2024-09-13T13:51:45.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.797+0000 7fd41fc60640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.797+0000 7fd41fc60640 10 monclient: monmap: 2024-09-13T13:51:45.802 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.802 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.802 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:45.799219+0000 2024-09-13T13:51:45.802 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:45.799219+0000 2024-09-13T13:51:45.803 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.803 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.803 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.803 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.803 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.803 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 1 Processor -- start 2024-09-13T13:51:45.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 1 -- start start 2024-09-13T13:51:45.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient: init 2024-09-13T13:51:45.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient: _add_conns ranks=[3,7,0,4,1,2,6,8,5] 2024-09-13T13:51:45.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient(hunting): picked mon.noname-g con 0x7fd41809c950 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:45.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient(hunting): picked mon.noname-i con 0x7fd418135b50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:45.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient(hunting): picked mon.noname-a con 0x7fd41813f660 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:45.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd418135f50 con 0x7fd41813f660 2024-09-13T13:51:45.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41809d860 con 0x7fd41809c950 2024-09-13T13:51:45.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41809db40 con 0x7fd418135b50 2024-09-13T13:51:45.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41fc60640 10 monclient(hunting): authenticate will time out at 785.428803s 2024-09-13T13:51:45.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41e1fd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fd41813f660 0x7fd418141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45532/0 (socket says 172.21.15.121:45532) 2024-09-13T13:51:45.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.798+0000 7fd41e1fd640 1 -- 172.21.15.121:0/962074984 learned_addr learned my addr 172.21.15.121:0/962074984 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41e1fd640 1 -- 172.21.15.121:0/962074984 >> v1:172.21.15.121:6789/0 conn(0x7fd41813f660 legacy=0x7fd418141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41d9fc640 1 -- 172.21.15.121:0/962074984 >> v1:172.21.15.153:6790/0 conn(0x7fd41809c950 legacy=0x7fd4180a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3828084482 0 0) 0x7fd418135f50 con 0x7fd41813f660 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): my global_id is 4410 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd404003520 con 0x7fd41813f660 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41d1fb640 1 -- 172.21.15.121:0/962074984 >> v1:172.21.15.153:6792/0 conn(0x7fd418135b50 legacy=0x7fd41813df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547065708 0 0) 0x7fd41809d860 con 0x7fd41809c950 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): my global_id is 4296 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd418135f50 con 0x7fd41809c950 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 807430369 0 0) 0x7fd41809db40 con 0x7fd418135b50 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient(hunting): my global_id is 4363 2024-09-13T13:51:45.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd41809d860 con 0x7fd418135b50 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1836824227 0 0) 0x7fd404003520 con 0x7fd41813f660 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 >> v1:172.21.15.153:6792/0 conn(0x7fd418135b50 legacy=0x7fd41813df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 >> v1:172.21.15.153:6790/0 conn(0x7fd41809c950 legacy=0x7fd4180a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 monclient: found mon.noname-a 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41809de20 con 0x7fd41813f660 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd410001e60 con 0x7fd41813f660 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: dump: 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.826 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.827 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41fc60640 5 monclient: authenticate success, global_id 4410 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41fc60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd4100020b0 con 0x7fd41813f660 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 1 -- 172.21.15.121:0/962074984 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd410004f60 con 0x7fd41813f660 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41c9fa640 10 monclient: dump: 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.829 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41fc60640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:45.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.801+0000 7fd41fc60640 4 set_mon_vals no callback set 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 10 monclient: shutdown 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 1 -- 172.21.15.121:0/962074984 >> v1:172.21.15.121:6789/0 conn(0x7fd41813f660 legacy=0x7fd418141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 1 -- 172.21.15.121:0/962074984 shutdown_connections 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 1 -- 172.21.15.121:0/962074984 >> 172.21.15.121:0/962074984 conn(0x7fd4180ab760 msgr2=0x7fd4180abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 1 -- 172.21.15.121:0/962074984 shutdown_connections 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 1 -- 172.21.15.121:0/962074984 wait complete. 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 10 monclient: build_initial_monmap 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.802+0000 7fd41fc60640 10 monclient: monmap: 2024-09-13T13:51:45.831 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:45.832 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 1 Processor -- start 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 1 -- start start 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient: init 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient: _add_conns ranks=[3,6,7,8,2,1,0,4,5] 2024-09-13T13:51:45.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient(hunting): picked mon.noname-g con 0x7fd41809c950 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient(hunting): picked mon.noname-d con 0x7fd418135b50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient(hunting): picked mon.noname-i con 0x7fd41813f660 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41813efd0 con 0x7fd41809c950 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd418142ea0 con 0x7fd418135b50 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41813de30 con 0x7fd41813f660 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41fc60640 10 monclient(hunting): authenticate will time out at 785.433585s 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41d1fb640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fd418135b50 0x7fd41813b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52368/0 (socket says 172.21.15.121:52368) 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.803+0000 7fd41d1fb640 1 -- 172.21.15.121:0/3661563174 learned_addr learned my addr 172.21.15.121:0/3661563174 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd41d1fb640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.121:6792/0 conn(0x7fd418135b50 legacy=0x7fd41813b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd41d9fc640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.153:6790/0 conn(0x7fd41809c950 legacy=0x7fd41813aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249830747 0 0) 0x7fd41809d860 con 0x7fd418135b50 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd40e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd40e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd40e7fc640 10 monclient(hunting): my global_id is 4326 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd41e1fd640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.153:6792/0 conn(0x7fd41813f660 legacy=0x7fd4181d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.805+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3fc003520 con 0x7fd418135b50 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3430543594 0 0) 0x7fd418135f50 con 0x7fd41809c950 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): my global_id is 4305 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd41809d860 con 0x7fd41809c950 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378852527 0 0) 0x7fd41813de30 con 0x7fd41813f660 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:45.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): my global_id is 4372 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd418135f50 con 0x7fd41813f660 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3861362818 0 0) 0x7fd3fc003520 con 0x7fd418135b50 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd41813de30 con 0x7fd418135b50 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd408003610 con 0x7fd418135b50 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.836 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3118464323 0 0) 0x7fd41809d860 con 0x7fd41809c950 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3fc003520 con 0x7fd41809c950 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd414003610 con 0x7fd41809c950 2024-09-13T13:51:45.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.838 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 759470791 0 0) 0x7fd418135f50 con 0x7fd41813f660 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd41809d860 con 0x7fd41813f660 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd4100016a0 con 0x7fd41813f660 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.839 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 762193305 0 0) 0x7fd41813de30 con 0x7fd418135b50 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.153:6792/0 conn(0x7fd41813f660 legacy=0x7fd4181d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.153:6790/0 conn(0x7fd41809c950 legacy=0x7fd41813aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 monclient: found mon.d 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41813e110 con 0x7fd418135b50 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 5 monclient: authenticate success, global_id 4326 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 10 monclient: _renew_subs 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd4180a5470 con 0x7fd418135b50 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 10 monclient: _renew_subs 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fd41813d170 con 0x7fd418135b50 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd408003be0 con 0x7fd418135b50 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fd408004110 con 0x7fd418135b50 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40ffff640 4 set_mon_vals no callback set 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.806+0000 7fd40e7fc640 10 monclient: dump: 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:45.842 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:45.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.807+0000 7fd41fc60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.807+0000 7fd41fc60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.807+0000 7fd41fc60640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.807+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd400005080 con 0x7fd418135b50 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.807+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fd4080042b0 con 0x7fd418135b50 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.807+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7fd40804e200 con 0x7fd418135b50 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.808+0000 7fd41d9fc640 10 monclient: get_auth_request con 0x7fd3fc040670 auth_method 0 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.812+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fd418142ea0 con 0x7fd418135b50 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.812+0000 7fd40e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:45.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.812+0000 7fd40e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:45.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.982+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd400000b90 con 0x7fd3fc040670 2024-09-13T13:51:45.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.983+0000 7fd40e7fc640 1 -- 172.21.15.121:0/3661563174 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34215 (unknown 2965378022 0 2437808176) 0x7fd400000b90 con 0x7fd3fc040670 2024-09-13T13:51:45.985 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:45.986 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:51:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.153:6812/1206688214 conn(0x7fd3fc040670 legacy=0x7fd3fc042b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 10 monclient: shutdown 2024-09-13T13:51:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 >> v1:172.21.15.121:6792/0 conn(0x7fd418135b50 legacy=0x7fd41813b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 shutdown_connections 2024-09-13T13:51:45.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 >> 172.21.15.121:0/3661563174 conn(0x7fd4180ab760 msgr2=0x7fd418134f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 shutdown_connections 2024-09-13T13:51:45.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:45.985+0000 7fd41fc60640 1 -- 172.21.15.121:0/3661563174 wait complete. 2024-09-13T13:51:45.997 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T13:51:45.219090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163312,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067728,"statfs":{"total":579820584960,"available":579653353472,"internally_reserved":0,"allocated":1146880,"data_stored":1385212,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.769933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470031+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:51:43.470031+0000","last_peered":"2024-09-13T13:51:43.470031+0000","last_clean":"2024-09-13T13:51:43.470031+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:51:43.470031+0000","last_undegraded":"2024-09-13T13:51:43.470031+0000","last_fullsized":"2024-09-13T13:51:43.470031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470070+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:51:43.470070+0000","last_peered":"2024-09-13T13:51:43.470070+0000","last_clean":"2024-09-13T13:51:43.470070+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:51:43.470070+0000","last_undegraded":"2024-09-13T13:51:43.470070+0000","last_fullsized":"2024-09-13T13:51:43.470070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470103+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:51:43.470103+0000","last_peered":"2024-09-13T13:51:43.470103+0000","last_clean":"2024-09-13T13:51:43.470103+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:51:43.470103+0000","last_undegraded":"2024-09-13T13:51:43.470103+0000","last_fullsized":"2024-09-13T13:51:43.470103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470118+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:51:43.470118+0000","last_peered":"2024-09-13T13:51:43.470118+0000","last_clean":"2024-09-13T13:51:43.470118+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:51:43.470118+0000","last_undegraded":"2024-09-13T13:51:43.470118+0000","last_fullsized":"2024-09-13T13:51:43.470118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.472412+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:51:43.472412+0000","last_peered":"2024-09-13T13:51:43.472412+0000","last_clean":"2024-09-13T13:51:43.472412+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:51:43.472412+0000","last_undegraded":"2024-09-13T13:51:43.472412+0000","last_fullsized":"2024-09-13T13:51:43.472412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.467703+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:51:43.467703+0000","last_peered":"2024-09-13T13:51:43.467703+0000","last_clean":"2024-09-13T13:51:43.467703+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:51:43.467703+0000","last_undegraded":"2024-09-13T13:51:43.467703+0000","last_fullsized":"2024-09-13T13:51:43.467703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470775+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:51:43.470775+0000","last_peered":"2024-09-13T13:51:43.470775+0000","last_clean":"2024-09-13T13:51:43.470775+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:51:43.470775+0000","last_undegraded":"2024-09-13T13:51:43.470775+0000","last_fullsized":"2024-09-13T13:51:43.470775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.471333+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:51:43.471333+0000","last_peered":"2024-09-13T13:51:43.471333+0000","last_clean":"2024-09-13T13:51:43.471333+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:51:43.471333+0000","last_undegraded":"2024-09-13T13:51:43.471333+0000","last_fullsized":"2024-09-13T13:51:43.471333+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470030+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:51:43.470030+0000","last_peered":"2024-09-13T13:51:43.470030+0000","last_clean":"2024-09-13T13:51:43.470030+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:51:43.470030+0000","last_undegraded":"2024-09-13T13:51:43.470030+0000","last_fullsized":"2024-09-13T13:51:43.470030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:45.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:46.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.093+0000 7f15685e8640 10 monclient: get_monmap_and_config 2024-09-13T13:51:46.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.093+0000 7f15685e8640 10 monclient: build_initial_monmap 2024-09-13T13:51:46.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.093+0000 7f15685e8640 10 monclient: monmap: 2024-09-13T13:51:46.095 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:46.095 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.095 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:46.095244+0000 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:46.095244+0000 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:46.096 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 1 Processor -- start 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 1 -- start start 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient: init 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient: _add_conns ranks=[2,8,0,7,4,6,1,5,3] 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient(hunting): picked mon.noname-b con 0x7f156009c950 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient(hunting): picked mon.noname-e con 0x7f1560135b50 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient(hunting): picked mon.noname-a con 0x7f156013f660 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1560135f50 con 0x7f156013f660 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f156009d860 con 0x7f156009c950 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f156009db40 con 0x7f1560135b50 2024-09-13T13:51:46.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:46.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f15685e8640 10 monclient(hunting): authenticate will time out at 785.724796s 2024-09-13T13:51:46.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f1566b85640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f156013f660 0x7f1560141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45534/0 (socket says 172.21.15.121:45534) 2024-09-13T13:51:46.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f1565b83640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f1560135b50 0x7f156013df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56632/0 (socket says 172.21.15.121:56632) 2024-09-13T13:51:46.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.094+0000 7f1566b85640 1 -- 172.21.15.121:0/2593124491 learned_addr learned my addr 172.21.15.121:0/2593124491 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1566384640 1 -- 172.21.15.121:0/2593124491 >> v1:172.21.15.121:6790/0 conn(0x7f156009c950 legacy=0x7f15600a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1566b85640 1 -- 172.21.15.121:0/2593124491 >> v1:172.21.15.121:6789/0 conn(0x7f156013f660 legacy=0x7f1560141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565b83640 1 -- 172.21.15.121:0/2593124491 >> v1:172.21.15.121:6793/0 conn(0x7f1560135b50 legacy=0x7f156013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 216583781 0 0) 0x7f1560135f50 con 0x7f156013f660 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): my global_id is 4419 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1548003520 con 0x7f156013f660 2024-09-13T13:51:46.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3419492413 0 0) 0x7f156009d860 con 0x7f156009c950 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): my global_id is 4295 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1560135f50 con 0x7f156009c950 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469617129 0 0) 0x7f156009db40 con 0x7f1560135b50 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient(hunting): my global_id is 4373 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f156009d860 con 0x7f1560135b50 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1292219846 0 0) 0x7f1548003520 con 0x7f156013f660 2024-09-13T13:51:46.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 >> v1:172.21.15.121:6793/0 conn(0x7f1560135b50 legacy=0x7f156013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 >> v1:172.21.15.121:6790/0 conn(0x7f156009c950 legacy=0x7f15600a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: _finish_hunting 0 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 monclient: found mon.noname-a 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156009de20 con 0x7f156013f660 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: _finish_auth 0 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: _check_auth_tickets 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1550001ae0 con 0x7f156013f660 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:46.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: dump: 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.106 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f15685e8640 5 monclient: authenticate success, global_id 4419 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f15685e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15500048a0 con 0x7f156013f660 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f15685e8640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 1 -- 172.21.15.121:0/2593124491 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1550004e10 con 0x7f156013f660 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f15685e8640 4 set_mon_vals no callback set 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.097+0000 7f1565382640 10 monclient: dump: 2024-09-13T13:51:46.107 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.108 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 10 monclient: shutdown 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 1 -- 172.21.15.121:0/2593124491 >> v1:172.21.15.121:6789/0 conn(0x7f156013f660 legacy=0x7f1560141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 1 -- 172.21.15.121:0/2593124491 shutdown_connections 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 1 -- 172.21.15.121:0/2593124491 >> 172.21.15.121:0/2593124491 conn(0x7f15600ab760 msgr2=0x7f15600abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 1 -- 172.21.15.121:0/2593124491 shutdown_connections 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 1 -- 172.21.15.121:0/2593124491 wait complete. 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 10 monclient: build_initial_monmap 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.098+0000 7f15685e8640 10 monclient: monmap: 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:46.109 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 1 Processor -- start 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 1 -- start start 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient: init 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient: _add_conns ranks=[6,7,8,4,0,3,2,5,1] 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient(hunting): picked mon.noname-d con 0x7f156009c950 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient(hunting): picked mon.noname-i con 0x7f1560135b50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient(hunting): picked mon.noname-e con 0x7f156013f660 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15601350e0 con 0x7f156009c950 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1560142e20 con 0x7f1560135b50 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f156013b430 con 0x7f156013f660 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f15685e8640 10 monclient(hunting): authenticate will time out at 785.729684s 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f1566384640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f156009c950 0x7f156013a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52374/0 (socket says 172.21.15.121:52374) 2024-09-13T13:51:46.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.099+0000 7f1566384640 1 -- 172.21.15.121:0/910965551 learned_addr learned my addr 172.21.15.121:0/910965551 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.101+0000 7f1566b85640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.121:6793/0 conn(0x7f156013f660 legacy=0x7f15601d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.101+0000 7f1566384640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.121:6792/0 conn(0x7f156009c950 legacy=0x7f156013a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453778415 0 0) 0x7f156013b430 con 0x7f156013f660 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): my global_id is 4382 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f154c003520 con 0x7f156013f660 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723467987 0 0) 0x7f15580018d0 con 0x7f156009c950 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): my global_id is 4335 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f156013b430 con 0x7f156009c950 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f1565b83640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.153:6792/0 conn(0x7f1560135b50 legacy=0x7f156013ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1072855978 0 0) 0x7f154c003520 con 0x7f156013f660 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15580018d0 con 0x7f156013f660 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1550001d30 con 0x7f156013f660 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): dump: 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1150711735 0 0) 0x7f156013b430 con 0x7f156009c950 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f154c003520 con 0x7f156009c950 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340078234 0 0) 0x7f1554001500 con 0x7f1560135b50 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): my global_id is 4381 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f156013b430 con 0x7f1560135b50 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1558003cd0 con 0x7f156009c950 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient(hunting): dump: 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1079130679 0 0) 0x7f15580018d0 con 0x7f156013f660 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.153:6792/0 conn(0x7f1560135b50 legacy=0x7f156013ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.121:6792/0 conn(0x7f156009c950 legacy=0x7f156013a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 monclient: found mon.e 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156013ecf0 con 0x7f156013f660 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: _finish_auth 0 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 5 monclient: authenticate success, global_id 4382 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 10 monclient: _renew_subs 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f156013e050 con 0x7f156013f660 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 10 monclient: _renew_subs 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f15600a53b0 con 0x7f156013f660 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15500048a0 con 0x7f156013f660 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1550005380 con 0x7f156013f660 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f1564b81640 4 set_mon_vals no callback set 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.102+0000 7f155effd640 10 monclient: dump: 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f15685e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f15685e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f15685e8640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1540005080 con 0x7f156013f660 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1550011a00 con 0x7f156013f660 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (unknown 3465900261 0 0) 0x7f155004e090 con 0x7f156013f660 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.103+0000 7f1566384640 10 monclient: get_auth_request con 0x7f154c0404a0 auth_method 0 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.108+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f15500201d0 con 0x7f156013f660 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.108+0000 7f155effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.108+0000 7f155effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.279+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1540000b90 con 0x7f154c0404a0 2024-09-13T13:51:46.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.280+0000 7f155effd640 1 -- 172.21.15.121:0/910965551 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34215 (unknown 2965378022 0 2437808176) 0x7f1540000b90 con 0x7f154c0404a0 2024-09-13T13:51:46.282 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:46.282 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.153:6812/1206688214 conn(0x7f154c0404a0 legacy=0x7f154c042960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 10 monclient: shutdown 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 >> v1:172.21.15.121:6793/0 conn(0x7f156013f660 legacy=0x7f15601d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 shutdown_connections 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 >> 172.21.15.121:0/910965551 conn(0x7f15600ab760 msgr2=0x7f15600a6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 shutdown_connections 2024-09-13T13:51:46.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.282+0000 7f15685e8640 1 -- 172.21.15.121:0/910965551 wait complete. 2024-09-13T13:51:46.294 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-13T13:51:45.219090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163312,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067728,"statfs":{"total":579820584960,"available":579653353472,"internally_reserved":0,"allocated":1146880,"data_stored":1385212,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.769933"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470031+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:51:43.470031+0000","last_peered":"2024-09-13T13:51:43.470031+0000","last_clean":"2024-09-13T13:51:43.470031+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:51:43.470031+0000","last_undegraded":"2024-09-13T13:51:43.470031+0000","last_fullsized":"2024-09-13T13:51:43.470031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470070+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:51:43.470070+0000","last_peered":"2024-09-13T13:51:43.470070+0000","last_clean":"2024-09-13T13:51:43.470070+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:51:43.470070+0000","last_undegraded":"2024-09-13T13:51:43.470070+0000","last_fullsized":"2024-09-13T13:51:43.470070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470103+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:51:43.470103+0000","last_peered":"2024-09-13T13:51:43.470103+0000","last_clean":"2024-09-13T13:51:43.470103+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:51:43.470103+0000","last_undegraded":"2024-09-13T13:51:43.470103+0000","last_fullsized":"2024-09-13T13:51:43.470103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470118+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:51:43.470118+0000","last_peered":"2024-09-13T13:51:43.470118+0000","last_clean":"2024-09-13T13:51:43.470118+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:51:43.470118+0000","last_undegraded":"2024-09-13T13:51:43.470118+0000","last_fullsized":"2024-09-13T13:51:43.470118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.472412+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:51:43.472412+0000","last_peered":"2024-09-13T13:51:43.472412+0000","last_clean":"2024-09-13T13:51:43.472412+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:51:43.472412+0000","last_undegraded":"2024-09-13T13:51:43.472412+0000","last_fullsized":"2024-09-13T13:51:43.472412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.467703+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:51:43.467703+0000","last_peered":"2024-09-13T13:51:43.467703+0000","last_clean":"2024-09-13T13:51:43.467703+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:51:43.467703+0000","last_undegraded":"2024-09-13T13:51:43.467703+0000","last_fullsized":"2024-09-13T13:51:43.467703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470775+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:51:43.470775+0000","last_peered":"2024-09-13T13:51:43.470775+0000","last_clean":"2024-09-13T13:51:43.470775+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:51:43.470775+0000","last_undegraded":"2024-09-13T13:51:43.470775+0000","last_fullsized":"2024-09-13T13:51:43.470775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.471333+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:51:43.471333+0000","last_peered":"2024-09-13T13:51:43.471333+0000","last_clean":"2024-09-13T13:51:43.471333+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:51:43.471333+0000","last_undegraded":"2024-09-13T13:51:43.471333+0000","last_fullsized":"2024-09-13T13:51:43.471333+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470030+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:51:43.470030+0000","last_peered":"2024-09-13T13:51:43.470030+0000","last_clean":"2024-09-13T13:51:43.470030+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:51:43.470030+0000","last_undegraded":"2024-09-13T13:51:43.470030+0000","last_fullsized":"2024-09-13T13:51:43.470030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:46.295 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:51:46.295 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T13:51:46.295 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T13:51:46.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T13:51:46.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.395+0000 7f684bb2b640 10 monclient: get_monmap_and_config 2024-09-13T13:51:46.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.395+0000 7f684bb2b640 10 monclient: build_initial_monmap 2024-09-13T13:51:46.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.395+0000 7f684bb2b640 10 monclient: monmap: 2024-09-13T13:51:46.397 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:46.397 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:46.397345+0000 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:46.397345+0000 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:46.398 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 1 Processor -- start 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 1 -- start start 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 10 monclient: init 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 10 monclient: _add_conns ranks=[6,1,2,3,0,8,4,5,7] 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 10 monclient(hunting): picked mon.noname-d con 0x7f6844138c30 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 10 monclient(hunting): picked mon.noname-f con 0x7f6844139410 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 10 monclient(hunting): picked mon.noname-b con 0x7f684413f6d0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6844139840 con 0x7f6844139410 2024-09-13T13:51:46.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.396+0000 7f684bb2b640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f684409db40 con 0x7f684413f6d0 2024-09-13T13:51:46.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.397+0000 7f684bb2b640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f684409de20 con 0x7f6844138c30 2024-09-13T13:51:46.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.397+0000 7f684bb2b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:46.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.397+0000 7f684bb2b640 10 monclient(hunting): authenticate will time out at 786.027018s 2024-09-13T13:51:46.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.397+0000 7f68498c7640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f6844138c30 0x7f68440aa680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52380/0 (socket says 172.21.15.121:52380) 2024-09-13T13:51:46.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.397+0000 7f68498c7640 1 -- 172.21.15.121:0/519881587 learned_addr learned my addr 172.21.15.121:0/519881587 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68498c7640 1 -- 172.21.15.121:0/519881587 >> v1:172.21.15.121:6792/0 conn(0x7f6844138c30 legacy=0x7f68440aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f684a0c8640 1 -- 172.21.15.121:0/519881587 >> v1:172.21.15.121:6790/0 conn(0x7f684413f6d0 legacy=0x7f6844141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700497056 0 0) 0x7f684409de20 con 0x7f6844138c30 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): my global_id is 4344 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6830003520 con 0x7f6844138c30 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900470136 0 0) 0x7f684409db40 con 0x7f684413f6d0 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient(hunting): my global_id is 4304 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f684409de20 con 0x7f684413f6d0 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68490c6640 1 -- 172.21.15.121:0/519881587 >> v1:172.21.15.153:6789/0 conn(0x7f6844139410 legacy=0x7f684413dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1859336024 0 0) 0x7f684409de20 con 0x7f684413f6d0 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 >> v1:172.21.15.121:6792/0 conn(0x7f6844138c30 legacy=0x7f68440aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 >> v1:172.21.15.153:6789/0 conn(0x7f6844139410 legacy=0x7f684413dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: _finish_hunting 0 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 monclient: found mon.noname-b 2024-09-13T13:51:46.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68440aad90 con 0x7f684413f6d0 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: _finish_auth 0 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: _check_auth_tickets 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f683c001e90 con 0x7f684413f6d0 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f684bb2b640 5 monclient: authenticate success, global_id 4304 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f684bb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: dump: 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.411 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f684bb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f683c003990 con 0x7f684413f6d0 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f684bb2b640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 1 -- 172.21.15.121:0/519881587 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f683c003ec0 con 0x7f684413f6d0 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f684bb2b640 4 set_mon_vals no callback set 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.399+0000 7f68488c5640 10 monclient: dump: 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 10 monclient: shutdown 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 1 -- 172.21.15.121:0/519881587 >> v1:172.21.15.121:6790/0 conn(0x7f684413f6d0 legacy=0x7f6844141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 1 -- 172.21.15.121:0/519881587 shutdown_connections 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 1 -- 172.21.15.121:0/519881587 >> 172.21.15.121:0/519881587 conn(0x7f68440ab800 msgr2=0x7f68440abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 1 -- 172.21.15.121:0/519881587 shutdown_connections 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 1 -- 172.21.15.121:0/519881587 wait complete. 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 10 monclient: build_initial_monmap 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.400+0000 7f684bb2b640 10 monclient: monmap: 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 1 Processor -- start 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 1 -- start start 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient: init 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient: _add_conns ranks=[7,8,1,3,4,6,0,2,5] 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient(hunting): picked mon.noname-i con 0x7f6844138c30 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:46.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient(hunting): picked mon.noname-e con 0x7f6844139410 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient(hunting): picked mon.noname-f con 0x7f684413f6d0 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f684413b980 con 0x7f684413f6d0 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6844142b70 con 0x7f6844138c30 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68440a8bc0 con 0x7f6844139410 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f684bb2b640 10 monclient(hunting): authenticate will time out at 786.031811s 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f68490c6640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f6844139410 0x7f684413ab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56654/0 (socket says 172.21.15.121:56654) 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.401+0000 7f68490c6640 1 -- 172.21.15.121:0/480194791 learned_addr learned my addr 172.21.15.121:0/480194791 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.403+0000 7f68490c6640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.121:6793/0 conn(0x7f6844139410 legacy=0x7f684413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050148525 0 0) 0x7f68440a8bc0 con 0x7f6844139410 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): my global_id is 4391 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6828003520 con 0x7f6844139410 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f68498c7640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.153:6792/0 conn(0x7f6844138c30 legacy=0x7f684413a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684a0c8640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.153:6789/0 conn(0x7f684413f6d0 legacy=0x7f68441d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2706118904 0 0) 0x7f6828003520 con 0x7f6844139410 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f68440a8bc0 con 0x7f6844139410 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6840003610 con 0x7f6844139410 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.418 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639088222 0 0) 0x7f684413b980 con 0x7f684413f6d0 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): my global_id is 4339 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6828003520 con 0x7f684413f6d0 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3158221512 0 0) 0x7f6844142b70 con 0x7f6844138c30 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient(hunting): my global_id is 4390 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f684413b980 con 0x7f6844138c30 2024-09-13T13:51:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 424740359 0 0) 0x7f68440a8bc0 con 0x7f6844139410 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.153:6792/0 conn(0x7f6844138c30 legacy=0x7f684413a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.153:6789/0 conn(0x7f684413f6d0 legacy=0x7f68441d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 monclient: found mon.e 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684413bf90 con 0x7f6844139410 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 5 monclient: authenticate success, global_id 4391 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 10 monclient: _renew_subs 2024-09-13T13:51:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f684413e2f0 con 0x7f6844139410 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 10 monclient: _renew_subs 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f68440a5230 con 0x7f6844139410 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6840003bd0 con 0x7f6844139410 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6840004100 con 0x7f6844139410 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683a7fc640 10 monclient: dump: 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.421 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.404+0000 7f683bfff640 4 set_mon_vals no callback set 2024-09-13T13:51:46.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f68400042a0 con 0x7f6844139410 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f684bb2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f684bb2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f684bb2b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f684409d390 con 0x7f6844139410 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7f684004e270 con 0x7f6844139410 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.405+0000 7f68498c7640 10 monclient: get_auth_request con 0x7f682803fd90 auth_method 0 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.410+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f6840017890 con 0x7f6844139410 2024-09-13T13:51:46.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.410+0000 7f683a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.410+0000 7f683a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f684bb2b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f684bb2b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f684bb2b640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f68440a5560 con 0x7f6844139410 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f683a7fc640 1 -- 172.21.15.121:0/480194791 <== mon.8 v1:172.21.15.121:6793/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f6840017190 con 0x7f6844139410 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f683a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.624+0000 7f683a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:46.626 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.153:6812/1206688214 conn(0x7f682803fd90 legacy=0x7f6828042250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 10 monclient: shutdown 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 >> v1:172.21.15.121:6793/0 conn(0x7f6844139410 legacy=0x7f684413ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 shutdown_connections 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 >> 172.21.15.121:0/480194791 conn(0x7f68440ab800 msgr2=0x7f684413ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 shutdown_connections 2024-09-13T13:51:46.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.626+0000 7f684bb2b640 1 -- 172.21.15.121:0/480194791 wait complete. 2024-09-13T13:51:46.638 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T13:51:46.638 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-13T13:51:46.704 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-13T13:51:46.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.801+0000 7fc3229d1640 10 monclient: get_monmap_and_config 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.801+0000 7fc3229d1640 10 monclient: build_initial_monmap 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.801+0000 7fc3229d1640 10 monclient: monmap: 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:46.803136+0000 2024-09-13T13:51:46.803 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:46.803136+0000 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:46.804 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 1 Processor -- start 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 1 -- start start 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient: init 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient: _add_conns ranks=[3,4,0,2,1,6,7,5,8] 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient(hunting): picked mon.noname-g con 0x7fc31c09c950 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient(hunting): picked mon.noname-c con 0x7fc31c135b50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient(hunting): picked mon.noname-a con 0x7fc31c13f660 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31c135f50 con 0x7fc31c13f660 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31c09d860 con 0x7fc31c09c950 2024-09-13T13:51:46.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31c09db40 con 0x7fc31c135b50 2024-09-13T13:51:46.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:46.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc3229d1640 10 monclient(hunting): authenticate will time out at 786.432769s 2024-09-13T13:51:46.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc31b7fe640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fc31c135b50 0x7fc31c13df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33808/0 (socket says 172.21.15.121:33808) 2024-09-13T13:51:46.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc320f6e640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc31c13f660 0x7fc31c141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45546/0 (socket says 172.21.15.121:45546) 2024-09-13T13:51:46.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.802+0000 7fc31b7fe640 1 -- 172.21.15.121:0/3801858172 learned_addr learned my addr 172.21.15.121:0/3801858172 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:46.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc320f6e640 1 -- 172.21.15.121:0/3801858172 >> v1:172.21.15.121:6789/0 conn(0x7fc31c13f660 legacy=0x7fc31c141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31b7fe640 1 -- 172.21.15.121:0/3801858172 >> v1:172.21.15.121:6791/0 conn(0x7fc31c135b50 legacy=0x7fc31c13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650384206 0 0) 0x7fc31c135f50 con 0x7fc31c13f660 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): my global_id is 4428 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc304003520 con 0x7fc31c13f660 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 904074499 0 0) 0x7fc31c09db40 con 0x7fc31c135b50 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient(hunting): my global_id is 4306 2024-09-13T13:51:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31bfff640 1 -- 172.21.15.121:0/3801858172 >> v1:172.21.15.153:6790/0 conn(0x7fc31c09c950 legacy=0x7fc31c0a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc31c135f50 con 0x7fc31c135b50 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 517428998 0 0) 0x7fc304003520 con 0x7fc31c13f660 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 >> v1:172.21.15.121:6791/0 conn(0x7fc31c135b50 legacy=0x7fc31c13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 >> v1:172.21.15.153:6790/0 conn(0x7fc31c09c950 legacy=0x7fc31c0a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 monclient: found mon.noname-a 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c09de20 con 0x7fc31c13f660 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: _finish_auth 0 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc314001e60 con 0x7fc31c13f660 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc3229d1640 5 monclient: authenticate success, global_id 4428 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc3229d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: dump: 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.814 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc3229d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc314004b60 con 0x7fc31c13f660 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc3229d1640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 1 -- 172.21.15.121:0/3801858172 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc3140050d0 con 0x7fc31c13f660 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc3229d1640 4 set_mon_vals no callback set 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.805+0000 7fc31affd640 10 monclient: dump: 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 10 monclient: shutdown 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 1 -- 172.21.15.121:0/3801858172 >> v1:172.21.15.121:6789/0 conn(0x7fc31c13f660 legacy=0x7fc31c141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 1 -- 172.21.15.121:0/3801858172 shutdown_connections 2024-09-13T13:51:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 1 -- 172.21.15.121:0/3801858172 >> 172.21.15.121:0/3801858172 conn(0x7fc31c0ab760 msgr2=0x7fc31c0abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 1 -- 172.21.15.121:0/3801858172 shutdown_connections 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 1 -- 172.21.15.121:0/3801858172 wait complete. 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 10 monclient: build_initial_monmap 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 10 monclient: monmap: 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.806+0000 7fc3229d1640 1 Processor -- start 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 1 -- start start 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient: init 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient: _add_conns ranks=[5,3,6,0,1,2,4,8,7] 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient(hunting): picked mon.noname-h con 0x7fc31c09c950 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient(hunting): picked mon.noname-g con 0x7fc31c135b50 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient(hunting): picked mon.noname-d con 0x7fc31c13f660 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:46.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31c135000 con 0x7fc31c135b50 2024-09-13T13:51:46.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31c13a9f0 con 0x7fc31c09c950 2024-09-13T13:51:46.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc31c137a60 con 0x7fc31c13f660 2024-09-13T13:51:46.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:46.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc3229d1640 10 monclient(hunting): authenticate will time out at 786.437640s 2024-09-13T13:51:46.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc320f6e640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fc31c13f660 0x7fc31c1d6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52390/0 (socket says 172.21.15.121:52390) 2024-09-13T13:51:46.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.807+0000 7fc320f6e640 1 -- 172.21.15.121:0/589130936 learned_addr learned my addr 172.21.15.121:0/589130936 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:46.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.809+0000 7fc320f6e640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.121:6792/0 conn(0x7fc31c13f660 legacy=0x7fc31c1d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.809+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577422421 0 0) 0x7fc31c137a60 con 0x7fc31c13f660 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.809+0000 7fc318ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.809+0000 7fc318ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.809+0000 7fc318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.809+0000 7fc318ff9640 10 monclient(hunting): my global_id is 4353 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc308003550 con 0x7fc31c13f660 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc31bfff640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.153:6791/0 conn(0x7fc31c09c950 legacy=0x7fc31c136c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc31b7fe640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.153:6790/0 conn(0x7fc31c135b50 legacy=0x7fc31c137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 870159484 0 0) 0x7fc308003550 con 0x7fc31c13f660 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc31c137a60 con 0x7fc31c13f660 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc3140020b0 con 0x7fc31c13f660 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:46.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): dump: 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.826 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244943494 0 0) 0x7fc31c13a9f0 con 0x7fc31c09c950 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): my global_id is 4307 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc308003550 con 0x7fc31c09c950 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1034998024 0 0) 0x7fc310001580 con 0x7fc31c135b50 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:46.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient(hunting): my global_id is 4323 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc31c13a9f0 con 0x7fc31c135b50 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1767338341 0 0) 0x7fc31c137a60 con 0x7fc31c13f660 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.153:6791/0 conn(0x7fc31c09c950 legacy=0x7fc31c136c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.153:6790/0 conn(0x7fc31c135b50 legacy=0x7fc31c137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 monclient: found mon.d 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c13b160 con 0x7fc31c13f660 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: _finish_auth 0 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:51:46.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 5 monclient: authenticate success, global_id 4353 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 10 monclient: _renew_subs 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c13a710 con 0x7fc31c13f660 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 10 monclient: _renew_subs 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fc31c138b40 con 0x7fc31c13f660 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc314005300 con 0x7fc31c13f660 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc31a7fc640 4 set_mon_vals no callback set 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc314004b60 con 0x7fc31c13f660 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:46.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.810+0000 7fc318ff9640 10 monclient: dump: 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:46.830 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.811+0000 7fc3229d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.811+0000 7fc3229d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.811+0000 7fc3229d1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.811+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fc314011cd0 con 0x7fc31c13f660 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.811+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2fc005080 con 0x7fc31c13f660 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.811+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7fc31404e520 con 0x7fc31c13f660 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.812+0000 7fc31bfff640 10 monclient: get_auth_request con 0x7fc30803ff90 auth_method 0 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.816+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fc3140174d0 con 0x7fc31c13f660 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.816+0000 7fc318ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:46.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.816+0000 7fc318ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:46.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.993+0000 7fc3229d1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:46.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.993+0000 7fc3229d1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:46.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.993+0000 7fc3229d1640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:46.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.993+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2fc005370 con 0x7fc31c13f660 2024-09-13T13:51:46.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.994+0000 7fc318ff9640 1 -- 172.21.15.121:0/589130936 <== mon.6 v1:172.21.15.121:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+10934 (unknown 898900276 0 1577119840) 0x7fc314020680 con 0x7fc31c13f660 2024-09-13T13:51:46.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.994+0000 7fc318ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:51:46.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.994+0000 7fc318ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:46.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:46.997 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":21,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:51:46.338516+0000","last_up_change":"2024-09-13T13:51:36.267919+0000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:51:35.278046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:51:40.243263+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":3060725185}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":3060725185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":3060725185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":3060725185}]},"public_addr":"172.21.15.121:6808/3060725185","cluster_addr":"172.21.15.121:6809/3060725185","heartbeat_back_addr":"172.21.15.121:6811/3060725185","heartbeat_front_addr":"172.21.15.121:6810/3060725185","state":["exists","up"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":1651307734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1651307734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1651307734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1651307734}]},"public_addr":"172.21.15.121:6800/1651307734","cluster_addr":"172.21.15.121:6801/1651307734","heartbeat_back_addr":"172.21.15.121:6803/1651307734","heartbeat_front_addr":"172.21.15.121:6802/1651307734","state":["exists","up"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2141781943}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2141781943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2141781943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2141781943}]},"public_addr":"172.21.15.121:6804/2141781943","cluster_addr":"172.21.15.121:6805/2141781943","heartbeat_back_addr":"172.21.15.121:6807/2141781943","heartbeat_front_addr":"172.21.15.121:6806/2141781943","state":["exists","up"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6808","nonce":934815702}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6809","nonce":934815702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6811","nonce":934815702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6810","nonce":934815702}]},"public_addr":"172.21.15.153:6808/934815702","cluster_addr":"172.21.15.153:6809/934815702","heartbeat_back_addr":"172.21.15.153:6811/934815702","heartbeat_front_addr":"172.21.15.153:6810/934815702","state":["exists","up"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6800","nonce":1083052486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6801","nonce":1083052486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6803","nonce":1083052486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6802","nonce":1083052486}]},"public_addr":"172.21.15.153:6800/1083052486","cluster_addr":"172.21.15.153:6801/1083052486","heartbeat_back_addr":"172.21.15.153:6803/1083052486","heartbeat_front_addr":"172.21.15.153:6802/1083052486","state":["exists","up"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6804","nonce":3215471673}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6805","nonce":3215471673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6807","nonce":3215471673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6806","nonce":3215471673}]},"public_addr":"172.21.15.153:6804/3215471673","cluster_addr":"172.21.15.153:6805/3215471673","heartbeat_back_addr":"172.21.15.153:6807/3215471673","heartbeat_front_addr":"172.21.15.153:6806/3215471673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.867807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.618229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.154306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.400091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:30.983001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.211095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:51:46.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.153:6812/1206688214 conn(0x7fc30803ff90 legacy=0x7fc308042450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 10 monclient: shutdown 2024-09-13T13:51:46.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:46.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 >> v1:172.21.15.121:6792/0 conn(0x7fc31c13f660 legacy=0x7fc31c1d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:46.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 shutdown_connections 2024-09-13T13:51:46.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 >> 172.21.15.121:0/589130936 conn(0x7fc31c0ab760 msgr2=0x7fc31c133e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:46.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 shutdown_connections 2024-09-13T13:51:46.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:46.996+0000 7fc3229d1640 1 -- 172.21.15.121:0/589130936 wait complete. 2024-09-13T13:51:47.010 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T13:51:35.278046+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-13T13:51:40.243263+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T13:51:47.010 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T13:51:47.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.108+0000 7ffb514b8640 10 monclient: get_monmap_and_config 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.108+0000 7ffb514b8640 10 monclient: build_initial_monmap 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.108+0000 7ffb514b8640 10 monclient: monmap: 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:47.110511+0000 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:47.110511+0000 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:47.111 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:47.112 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 1 Processor -- start 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 1 -- start start 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient: init 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient: _add_conns ranks=[4,5,6,2,0,3,7,1,8] 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4c0a6160 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient(hunting): picked mon.noname-h con 0x7ffb4c09d150 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient(hunting): picked mon.noname-d con 0x7ffb4c0a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb4c09e210 con 0x7ffb4c0a6160 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb4c0a7ef0 con 0x7ffb4c09d150 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb4c09d970 con 0x7ffb4c0a6c50 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.109+0000 7ffb514b8640 10 monclient(hunting): authenticate will time out at 786.739909s 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.110+0000 7ffb4b7fe640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7ffb4c0a6c50 0x7ffb4c141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52394/0 (socket says 172.21.15.121:52394) 2024-09-13T13:51:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.110+0000 7ffb4b7fe640 1 -- 172.21.15.121:0/2372336483 learned_addr learned my addr 172.21.15.121:0/2372336483 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.110+0000 7ffb4affd640 1 --1- 172.21.15.121:0/2372336483 >> v1:172.21.15.121:6791/0 conn(0x7ffb4c0a6160 0x7ffb4c0a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33810/0 (socket says 172.21.15.121:33810) 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4affd640 1 -- 172.21.15.121:0/2372336483 >> v1:172.21.15.121:6791/0 conn(0x7ffb4c0a6160 legacy=0x7ffb4c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4b7fe640 1 -- 172.21.15.121:0/2372336483 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c0a6c50 legacy=0x7ffb4c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3974991714 0 0) 0x7ffb4c09e210 con 0x7ffb4c0a6160 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 4315 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb38003520 con 0x7ffb4c0a6160 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943710891 0 0) 0x7ffb4c09d970 con 0x7ffb4c0a6c50 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 4362 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb4c09e210 con 0x7ffb4c0a6c50 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb427fc640 1 -- 172.21.15.121:0/2372336483 >> v1:172.21.15.153:6791/0 conn(0x7ffb4c09d150 legacy=0x7ffb4c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2884738740 0 0) 0x7ffb38003520 con 0x7ffb4c0a6160 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c0a6c50 legacy=0x7ffb4c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 >> v1:172.21.15.153:6791/0 conn(0x7ffb4c09d150 legacy=0x7ffb4c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 monclient: found mon.noname-c 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:47.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4c09e630 con 0x7ffb4c0a6160 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb514b8640 5 monclient: authenticate success, global_id 4315 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb514b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb3c001ae0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: dump: 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.123 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.124 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb514b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4affd640 1 --1- 172.21.15.121:0/2372336483 >> v1:172.21.15.121:6791/0 conn(0x7ffb4c0a6160 0x7ffb4c0a6540 :-1 s=OPENED pgs=64 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7ffb3c004d50 mon_map magic: 0 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb3c0047e0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb514b8640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.112+0000 7ffb514b8640 4 set_mon_vals no callback set 2024-09-13T13:51:47.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 10 monclient: shutdown 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 1 -- 172.21.15.121:0/2372336483 >> v1:172.21.15.121:6791/0 conn(0x7ffb4c0a6160 legacy=0x7ffb4c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb4a7fc640 1 -- 172.21.15.121:0/2372336483 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb3c004d50 con 0x7ffb4c0a6160 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb4a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 1 -- 172.21.15.121:0/2372336483 shutdown_connections 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 1 -- 172.21.15.121:0/2372336483 >> 172.21.15.121:0/2372336483 conn(0x7ffb4c0ab940 msgr2=0x7ffb4c0abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 1 -- 172.21.15.121:0/2372336483 shutdown_connections 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 1 -- 172.21.15.121:0/2372336483 wait complete. 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 10 monclient: build_initial_monmap 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.113+0000 7ffb514b8640 10 monclient: monmap: 2024-09-13T13:51:47.126 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:47.127 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 1 Processor -- start 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 1 -- start start 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient: init 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient: _add_conns ranks=[6,2,3,0,7,5,4,1,8] 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient(hunting): picked mon.noname-d con 0x7ffb4c09d150 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb4c0a6160 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient(hunting): picked mon.noname-g con 0x7ffb4c0a6c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:47.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb4c129070 con 0x7ffb4c0a6160 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb4c129350 con 0x7ffb4c0a6c50 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb4c1287c0 con 0x7ffb4c09d150 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb514b8640 10 monclient(hunting): authenticate will time out at 786.744878s 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb4affd640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 0x7ffb4c125990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52404/0 (socket says 172.21.15.121:52404) 2024-09-13T13:51:47.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.114+0000 7ffb4affd640 1 -- 172.21.15.121:0/554409707 learned_addr learned my addr 172.21.15.121:0/554409707 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb427fc640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6790/0 conn(0x7ffb4c0a6160 legacy=0x7ffb4c1260a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb4affd640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 legacy=0x7ffb4c125990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2214644171 0 0) 0x7ffb4c1287c0 con 0x7ffb4c09d150 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): my global_id is 4371 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb2c003520 con 0x7ffb4c09d150 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2151815041 0 0) 0x7ffb4c129070 con 0x7ffb4c0a6160 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): my global_id is 4313 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb4c1287c0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb4affd640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 legacy=0x7ffb4c125990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb4affd640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 legacy=0x7ffb4c125990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:47.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb4affd640 1 --1- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 0x7ffb4c125990 :-1 s=OPENED pgs=71 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb4affd640 1 --1- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 0x7ffb4c125990 :-1 s=OPENED pgs=71 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6792/0 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb4b7fe640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.153:6790/0 conn(0x7ffb4c0a6c50 legacy=0x7ffb4c1d6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1371394649 0 0) 0x7ffb4c1287c0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb4c129070 con 0x7ffb4c0a6160 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb34003610 con 0x7ffb4c0a6160 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): dump: 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.135 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.136 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609358201 0 0) 0x7ffb4c129350 con 0x7ffb4c0a6c50 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient(hunting): my global_id is 4332 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb4c1287c0 con 0x7ffb4c0a6c50 2024-09-13T13:51:47.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2813561752 0 0) 0x7ffb4c129070 con 0x7ffb4c0a6160 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6792/0 conn(0x7ffb4c09d150 legacy=0x7ffb4c125990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.153:6790/0 conn(0x7ffb4c0a6c50 legacy=0x7ffb4c1d6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 monclient: found mon.b 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4c1267b0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: _finish_auth 0 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:47.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 5 monclient: authenticate success, global_id 4313 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 10 monclient: _renew_subs 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffb4c129630 con 0x7ffb4c0a6160 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 10 monclient: _renew_subs 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ffb4c1280a0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb34003ae0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb34004010 con 0x7ffb4c0a6160 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb49ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.117+0000 7ffb43fff640 10 monclient: dump: 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.140 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7ffb340042e0 con 0x7ffb4c0a6160 2024-09-13T13:51:47.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb514b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb514b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb514b8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb30005080 con 0x7ffb4c0a6160 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7ffb3404e200 con 0x7ffb4c0a6160 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.118+0000 7ffb4affd640 10 monclient: get_auth_request con 0x7ffb2c03fcd0 auth_method 0 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.123+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7ffb34017800 con 0x7ffb4c0a6160 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.123+0000 7ffb43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.123+0000 7ffb43fff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:47.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb514b8640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T13:51:47.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb514b8640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T13:51:47.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb514b8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:47.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffb30005c40 con 0x7ffb4c0a6160 2024-09-13T13:51:47.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb43fff640 1 -- 172.21.15.121:0/554409707 <== mon.2 v1:172.21.15.121:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (unknown 3303410134 0 2170607528) 0x7ffb34017100 con 0x7ffb4c0a6160 2024-09-13T13:51:47.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-13T13:51:47.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.295+0000 7ffb43fff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:47.298 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.153:6812/1206688214 conn(0x7ffb2c03fcd0 legacy=0x7ffb2c042190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 10 monclient: shutdown 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 >> v1:172.21.15.121:6790/0 conn(0x7ffb4c0a6160 legacy=0x7ffb4c1260a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 shutdown_connections 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 >> 172.21.15.121:0/554409707 conn(0x7ffb4c0ab940 msgr2=0x7ffb4c142aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 shutdown_connections 2024-09-13T13:51:47.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.298+0000 7ffb514b8640 1 -- 172.21.15.121:0/554409707 wait complete. 2024-09-13T13:51:47.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-13T13:51:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.412+0000 7fcc49f64640 10 monclient: get_monmap_and_config 2024-09-13T13:51:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.412+0000 7fcc49f64640 10 monclient: build_initial_monmap 2024-09-13T13:51:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.412+0000 7fcc49f64640 10 monclient: monmap: 2024-09-13T13:51:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:47.414 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:47.414316+0000 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:47.414316+0000 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:47.415 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 1 Processor -- start 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 1 -- start start 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient: init 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient: _add_conns ranks=[2,0,5,1,4,3,6,8,7] 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient(hunting): picked mon.noname-b con 0x7fcc44138bd0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient(hunting): picked mon.noname-a con 0x7fcc44138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient(hunting): picked mon.noname-h con 0x7fcc44139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc4409e100 con 0x7fcc44138fb0 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc4409db40 con 0x7fcc44138bd0 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc4409de20 con 0x7fcc44139420 2024-09-13T13:51:47.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc49f64640 10 monclient(hunting): authenticate will time out at 787.043870s 2024-09-13T13:51:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc42ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcc44138fb0 0x7fcc4413df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45552/0 (socket says 172.21.15.121:45552) 2024-09-13T13:51:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc437fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fcc44138bd0 0x7fcc440a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59186/0 (socket says 172.21.15.121:59186) 2024-09-13T13:51:47.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.413+0000 7fcc42ffd640 1 -- 172.21.15.121:0/2522345257 learned_addr learned my addr 172.21.15.121:0/2522345257 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:47.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc437fe640 1 -- 172.21.15.121:0/2522345257 >> v1:172.21.15.121:6790/0 conn(0x7fcc44138bd0 legacy=0x7fcc440a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc42ffd640 1 -- 172.21.15.121:0/2522345257 >> v1:172.21.15.121:6789/0 conn(0x7fcc44138fb0 legacy=0x7fcc4413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365954856 0 0) 0x7fcc4409db40 con 0x7fcc44138bd0 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): my global_id is 4322 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc2c003520 con 0x7fcc44138bd0 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038483766 0 0) 0x7fcc4409e100 con 0x7fcc44138fb0 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient(hunting): my global_id is 4437 2024-09-13T13:51:47.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc4409db40 con 0x7fcc44138fb0 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc43fff640 1 -- 172.21.15.121:0/2522345257 >> v1:172.21.15.153:6791/0 conn(0x7fcc44139420 legacy=0x7fcc44141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2770424029 0 0) 0x7fcc2c003520 con 0x7fcc44138bd0 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 >> v1:172.21.15.153:6791/0 conn(0x7fcc44139420 legacy=0x7fcc44141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 >> v1:172.21.15.121:6789/0 conn(0x7fcc44138fb0 legacy=0x7fcc4413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 monclient: found mon.noname-b 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc440a5da0 con 0x7fcc44138bd0 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:47.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcc34001ae0 con 0x7fcc44138bd0 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc49f64640 5 monclient: authenticate success, global_id 4322 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc49f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: dump: 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.428 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc49f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:47.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3896206413 0 0) 0x7fcc4409db40 con 0x7fcc44138fb0 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcc38001ac0 con 0x7fcc44138fb0 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc340047e0 con 0x7fcc44138bd0 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 1 -- 172.21.15.121:0/2522345257 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcc34004d90 con 0x7fcc44138bd0 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc49f64640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc49f64640 4 set_mon_vals no callback set 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.416+0000 7fcc427fc640 10 monclient: dump: 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.430 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 10 monclient: shutdown 2024-09-13T13:51:47.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 1 -- 172.21.15.121:0/2522345257 >> v1:172.21.15.121:6790/0 conn(0x7fcc44138bd0 legacy=0x7fcc440a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 1 -- 172.21.15.121:0/2522345257 shutdown_connections 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 1 -- 172.21.15.121:0/2522345257 >> 172.21.15.121:0/2522345257 conn(0x7fcc440ab800 msgr2=0x7fcc440abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 1 -- 172.21.15.121:0/2522345257 shutdown_connections 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 1 -- 172.21.15.121:0/2522345257 wait complete. 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 10 monclient: build_initial_monmap 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.417+0000 7fcc49f64640 10 monclient: monmap: 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:47.432 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:47.433 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 1 Processor -- start 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 1 -- start start 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient: init 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient: _add_conns ranks=[2,0,5,7,4,6,3,1,8] 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient(hunting): picked mon.noname-b con 0x7fcc44138bd0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient(hunting): picked mon.noname-a con 0x7fcc44138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient(hunting): picked mon.noname-h con 0x7fcc44139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc44131690 con 0x7fcc44138fb0 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc44142a40 con 0x7fcc44138bd0 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc4412fbd0 con 0x7fcc44139420 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc49f64640 10 monclient(hunting): authenticate will time out at 787.048768s 2024-09-13T13:51:47.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc437fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fcc44138bd0 0x7fcc44141c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59202/0 (socket says 172.21.15.121:59202) 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.418+0000 7fcc437fe640 1 -- 172.21.15.121:0/99956073 learned_addr learned my addr 172.21.15.121:0/99956073 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.420+0000 7fcc437fe640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.121:6790/0 conn(0x7fcc44138bd0 legacy=0x7fcc44141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.420+0000 7fcc42ffd640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.121:6789/0 conn(0x7fcc44138fb0 legacy=0x7fcc44142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016134914 0 0) 0x7fcc44142a40 con 0x7fcc44138bd0 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): my global_id is 4331 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc30003520 con 0x7fcc44138bd0 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532381068 0 0) 0x7fcc38001580 con 0x7fcc44138fb0 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): my global_id is 4446 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc43fff640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.153:6791/0 conn(0x7fcc44139420 legacy=0x7fcc441d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc44142a40 con 0x7fcc44138fb0 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904494663 0 0) 0x7fcc4412fbd0 con 0x7fcc44139420 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): my global_id is 4334 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc38001580 con 0x7fcc44139420 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1967981836 0 0) 0x7fcc44142a40 con 0x7fcc44138fb0 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc4412fbd0 con 0x7fcc44138fb0 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2530901223 0 0) 0x7fcc30003520 con 0x7fcc44138bd0 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc44142a40 con 0x7fcc44138bd0 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcc34004cd0 con 0x7fcc44138bd0 2024-09-13T13:51:47.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): dump: 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.437 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcc38001320 con 0x7fcc44138fb0 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient(hunting): dump: 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.438 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2722315823 0 0) 0x7fcc4412fbd0 con 0x7fcc44138fb0 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.153:6791/0 conn(0x7fcc44139420 legacy=0x7fcc441d6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.121:6790/0 conn(0x7fcc44138bd0 legacy=0x7fcc44141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: _finish_hunting 0 2024-09-13T13:51:47.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 monclient: found mon.a 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc44131f30 con 0x7fcc44138fb0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: _finish_auth 0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: _check_auth_tickets 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 5 monclient: authenticate success, global_id 4446 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 10 monclient: _renew_subs 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc44131360 con 0x7fcc44138fb0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 10 monclient: _renew_subs 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc441326c0 con 0x7fcc44138fb0 2024-09-13T13:51:47.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc38004dd0 con 0x7fcc44138fb0 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcc38005300 con 0x7fcc44138fb0 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc41ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.421+0000 7fcc2bfff640 10 monclient: dump: 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.441 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fcc380055a0 con 0x7fcc44138fb0 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7fcc3804e3b0 con 0x7fcc44138fb0 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc49f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc49f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc49f64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:47.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc24005080 con 0x7fcc44138fb0 2024-09-13T13:51:47.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.422+0000 7fcc437fe640 10 monclient: get_auth_request con 0x7fcc30040160 auth_method 0 2024-09-13T13:51:47.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.427+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fcc44131690 con 0x7fcc44138fb0 2024-09-13T13:51:47.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.427+0000 7fcc2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.427+0000 7fcc2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:47.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc49f64640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T13:51:47.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc49f64640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T13:51:47.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc49f64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:47.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fcc24005b80 con 0x7fcc44138fb0 2024-09-13T13:51:47.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc2bfff640 1 -- 172.21.15.121:0/99956073 <== mon.0 v1:172.21.15.121:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) ==== 92+0+10 (unknown 4161178158 0 265718887) 0x7fcc38017950 con 0x7fcc44138fb0 2024-09-13T13:51:47.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-13T13:51:47.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.605+0000 7fcc2bfff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:47.608 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 8 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.153:6812/1206688214 conn(0x7fcc30040160 legacy=0x7fcc30042620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 10 monclient: shutdown 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 >> v1:172.21.15.121:6789/0 conn(0x7fcc44138fb0 legacy=0x7fcc44142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 shutdown_connections 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 >> 172.21.15.121:0/99956073 conn(0x7fcc440ab800 msgr2=0x7fcc44137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 shutdown_connections 2024-09-13T13:51:47.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.608+0000 7fcc49f64640 1 -- 172.21.15.121:0/99956073 wait complete. 2024-09-13T13:51:47.620 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-13T13:51:47.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:47.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.715+0000 7f9546abf640 10 monclient: get_monmap_and_config 2024-09-13T13:51:47.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.715+0000 7f9546abf640 10 monclient: build_initial_monmap 2024-09-13T13:51:47.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.715+0000 7f9546abf640 10 monclient: monmap: 2024-09-13T13:51:47.717 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:47.717 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.717 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:47.717179+0000 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:47.717179+0000 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:47.718 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 1 Processor -- start 2024-09-13T13:51:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 1 -- start start 2024-09-13T13:51:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient: init 2024-09-13T13:51:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient: _add_conns ranks=[4,6,5,0,1,3,7,8,2] 2024-09-13T13:51:47.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient(hunting): picked mon.noname-c con 0x7f954009c950 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient(hunting): picked mon.noname-d con 0x7f9540135b50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient(hunting): picked mon.noname-h con 0x7f954013f660 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9540135f50 con 0x7f954009c950 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954009d860 con 0x7f954013f660 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954009db40 con 0x7f9540135b50 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f9546abf640 10 monclient(hunting): authenticate will time out at 787.346725s 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f954485b640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 0x7f95400a7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33818/0 (socket says 172.21.15.121:33818) 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f953ffff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f9540135b50 0x7f954013df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52410/0 (socket says 172.21.15.121:52410) 2024-09-13T13:51:47.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.716+0000 7f954485b640 1 -- 172.21.15.121:0/1545918065 learned_addr learned my addr 172.21.15.121:0/1545918065 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:47.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.718+0000 7f954485b640 1 -- 172.21.15.121:0/1545918065 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 legacy=0x7f95400a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953ffff640 1 -- 172.21.15.121:0/1545918065 >> v1:172.21.15.121:6792/0 conn(0x7f9540135b50 legacy=0x7f954013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f954505c640 1 -- 172.21.15.121:0/1545918065 >> v1:172.21.15.153:6791/0 conn(0x7f954013f660 legacy=0x7f9540141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3915369081 0 0) 0x7f954009db40 con 0x7f9540135b50 2024-09-13T13:51:47.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): my global_id is 4380 2024-09-13T13:51:47.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f952c003520 con 0x7f9540135b50 2024-09-13T13:51:47.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961243455 0 0) 0x7f9540135f50 con 0x7f954009c950 2024-09-13T13:51:47.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): my global_id is 4324 2024-09-13T13:51:47.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f954009db40 con 0x7f954009c950 2024-09-13T13:51:47.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4062684321 0 0) 0x7f954009d860 con 0x7f954013f660 2024-09-13T13:51:47.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient(hunting): my global_id is 4343 2024-09-13T13:51:47.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9540135f50 con 0x7f954013f660 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2423297272 0 0) 0x7f952c003520 con 0x7f9540135b50 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 >> v1:172.21.15.153:6791/0 conn(0x7f954013f660 legacy=0x7f9540141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 legacy=0x7f95400a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 monclient: found mon.noname-d 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:47.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954009de20 con 0x7f9540135b50 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9530001e60 con 0x7f9540135b50 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f9546abf640 5 monclient: authenticate success, global_id 4380 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f9546abf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:47.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: dump: 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.740 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.742 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f9546abf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9530004b60 con 0x7f9540135b50 2024-09-13T13:51:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 1 -- 172.21.15.121:0/1545918065 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f95300050d0 con 0x7f9540135b50 2024-09-13T13:51:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f9546abf640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f9546abf640 4 set_mon_vals no callback set 2024-09-13T13:51:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.719+0000 7f953f7fe640 10 monclient: dump: 2024-09-13T13:51:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.744 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 10 monclient: shutdown 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 1 -- 172.21.15.121:0/1545918065 >> v1:172.21.15.121:6792/0 conn(0x7f9540135b50 legacy=0x7f954013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 1 -- 172.21.15.121:0/1545918065 shutdown_connections 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 1 -- 172.21.15.121:0/1545918065 >> 172.21.15.121:0/1545918065 conn(0x7f95400ab760 msgr2=0x7f95400abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 1 -- 172.21.15.121:0/1545918065 shutdown_connections 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 1 -- 172.21.15.121:0/1545918065 wait complete. 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 10 monclient: build_initial_monmap 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.720+0000 7f9546abf640 10 monclient: monmap: 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:47.747 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 1 Processor -- start 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 1 -- start start 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient: init 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient: _add_conns ranks=[4,1,3,5,2,6,0,8,7] 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient(hunting): picked mon.noname-c con 0x7f954009c950 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient(hunting): picked mon.noname-f con 0x7f9540135b50 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient(hunting): picked mon.noname-g con 0x7f954013f660 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954013e4d0 con 0x7f9540135b50 2024-09-13T13:51:47.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954013ccd0 con 0x7f954013f660 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9540135350 con 0x7f954009c950 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f9546abf640 10 monclient(hunting): authenticate will time out at 787.351844s 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f954485b640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 0x7f954013a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33822/0 (socket says 172.21.15.121:33822) 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.721+0000 7f954485b640 1 -- 172.21.15.121:0/2271641250 learned_addr learned my addr 172.21.15.121:0/2271641250 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f954485b640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 legacy=0x7f954013a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f954485b640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 legacy=0x7f954013a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f954485b640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 legacy=0x7f954013a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f954485b640 1 --1- 172.21.15.121:0/2271641250 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 0x7f954013a770 :-1 s=OPENED pgs=67 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f954485b640 1 --1- 172.21.15.121:0/2271641250 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 0x7f954013a770 :-1 s=OPENED pgs=67 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6791/0 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f954505c640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.153:6790/0 conn(0x7f954013f660 legacy=0x7f954013b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953ffff640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.153:6789/0 conn(0x7f9540135b50 legacy=0x7f954013ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3685890674 0 0) 0x7f954013ccd0 con 0x7f954013f660 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): my global_id is 4341 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9520003550 con 0x7f954013f660 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3949086634 0 0) 0x7f954013e4d0 con 0x7f9540135b50 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): my global_id is 4348 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f954013ccd0 con 0x7f9540135b50 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2928190740 0 0) 0x7f9520003550 con 0x7f954013f660 2024-09-13T13:51:47.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f954013e4d0 con 0x7f954013f660 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9538001d30 con 0x7f954013f660 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): dump: 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.751 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4046725949 0 0) 0x7f954013e4d0 con 0x7f954013f660 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.121:6791/0 conn(0x7f954009c950 legacy=0x7f954013a770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.153:6789/0 conn(0x7f9540135b50 legacy=0x7f954013ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient: _finish_hunting 0 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 monclient: found mon.g 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:47.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954013edb0 con 0x7f954013f660 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient: _finish_auth 0 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 10 monclient: _check_auth_tickets 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.724+0000 7f953d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 5 monclient: authenticate success, global_id 4341 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 10 monclient: _renew_subs 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f954013c170 con 0x7f954013f660 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 10 monclient: _renew_subs 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:47.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f95401d70c0 con 0x7f954013f660 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9538003a40 con 0x7f954013f660 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9538004030 con 0x7f954013f660 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953effd640 4 set_mon_vals no callback set 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953d7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.725+0000 7f953d7fa640 10 monclient: dump: 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:47.754 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:47.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.726+0000 7f9546abf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.726+0000 7f9546abf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.726+0000 7f9546abf640 10 monclient: _send_mon_message to mon.g at v1:172.21.15.153:6790/0 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.726+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9524005080 con 0x7f954013f660 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.726+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f95380041d0 con 0x7f954013f660 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.726+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7f953804e0c0 con 0x7f954013f660 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.727+0000 7f954485b640 10 monclient: get_auth_request con 0x7f952003f9c0 auth_method 0 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.731+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mon.3 v1:172.21.15.153:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f953801b770 con 0x7f954013f660 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.731+0000 7f953d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:47.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.731+0000 7f953d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:47.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.904+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9524000b90 con 0x7f952003f9c0 2024-09-13T13:51:47.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.905+0000 7f953d7fa640 1 -- 172.21.15.121:0/2271641250 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34163 (unknown 2965378022 0 3406736524) 0x7f9524000b90 con 0x7f952003f9c0 2024-09-13T13:51:47.907 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:47.908 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.153:6812/1206688214 conn(0x7f952003f9c0 legacy=0x7f9520041e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 10 monclient: shutdown 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 >> v1:172.21.15.153:6790/0 conn(0x7f954013f660 legacy=0x7f954013b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 shutdown_connections 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 >> 172.21.15.121:0/2271641250 conn(0x7f95400ab760 msgr2=0x7f95400a6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 shutdown_connections 2024-09-13T13:51:47.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:47.907+0000 7f9546abf640 1 -- 172.21.15.121:0/2271641250 wait complete. 2024-09-13T13:51:47.919 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-13T13:51:47.219537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163312,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067728,"statfs":{"total":579820584960,"available":579653353472,"internally_reserved":0,"allocated":1146880,"data_stored":1385212,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.771462"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470031+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:51:43.470031+0000","last_peered":"2024-09-13T13:51:43.470031+0000","last_clean":"2024-09-13T13:51:43.470031+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:51:43.470031+0000","last_undegraded":"2024-09-13T13:51:43.470031+0000","last_fullsized":"2024-09-13T13:51:43.470031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.347416+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:51:46.347416+0000","last_peered":"2024-09-13T13:51:46.347416+0000","last_clean":"2024-09-13T13:51:46.347416+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:51:46.347416+0000","last_undegraded":"2024-09-13T13:51:46.347416+0000","last_fullsized":"2024-09-13T13:51:46.347416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470103+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:51:43.470103+0000","last_peered":"2024-09-13T13:51:43.470103+0000","last_clean":"2024-09-13T13:51:43.470103+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:51:43.470103+0000","last_undegraded":"2024-09-13T13:51:43.470103+0000","last_fullsized":"2024-09-13T13:51:43.470103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.347665+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:51:46.347665+0000","last_peered":"2024-09-13T13:51:46.347665+0000","last_clean":"2024-09-13T13:51:46.347665+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:51:46.347665+0000","last_undegraded":"2024-09-13T13:51:46.347665+0000","last_fullsized":"2024-09-13T13:51:46.347665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.472412+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:51:43.472412+0000","last_peered":"2024-09-13T13:51:43.472412+0000","last_clean":"2024-09-13T13:51:43.472412+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:51:43.472412+0000","last_undegraded":"2024-09-13T13:51:43.472412+0000","last_fullsized":"2024-09-13T13:51:43.472412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.467703+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:51:43.467703+0000","last_peered":"2024-09-13T13:51:43.467703+0000","last_clean":"2024-09-13T13:51:43.467703+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:51:43.467703+0000","last_undegraded":"2024-09-13T13:51:43.467703+0000","last_fullsized":"2024-09-13T13:51:43.467703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470775+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:51:43.470775+0000","last_peered":"2024-09-13T13:51:43.470775+0000","last_clean":"2024-09-13T13:51:43.470775+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:51:43.470775+0000","last_undegraded":"2024-09-13T13:51:43.470775+0000","last_fullsized":"2024-09-13T13:51:43.470775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.471333+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:51:43.471333+0000","last_peered":"2024-09-13T13:51:43.471333+0000","last_clean":"2024-09-13T13:51:43.471333+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:51:43.471333+0000","last_undegraded":"2024-09-13T13:51:43.471333+0000","last_fullsized":"2024-09-13T13:51:43.471333+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470030+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:51:43.470030+0000","last_peered":"2024-09-13T13:51:43.470030+0000","last_clean":"2024-09-13T13:51:43.470030+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:51:43.470030+0000","last_undegraded":"2024-09-13T13:51:43.470030+0000","last_fullsized":"2024-09-13T13:51:43.470030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:47.920 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.015+0000 7f5577216640 10 monclient: get_monmap_and_config 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.015+0000 7f5577216640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.015+0000 7f5577216640 10 monclient: monmap: 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:48.016873+0000 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:48.016873+0000 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.018 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.015+0000 7f5577216640 1 Processor -- start 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.015+0000 7f5577216640 1 -- start start 2024-09-13T13:51:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.015+0000 7f5577216640 10 monclient: init 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient: _add_conns ranks=[8,4,5,0,1,2,3,7,6] 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient(hunting): picked mon.noname-e con 0x7f557009c950 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient(hunting): picked mon.noname-c con 0x7f5570135b50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient(hunting): picked mon.noname-h con 0x7f557013f660 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5570135f50 con 0x7f5570135b50 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557009d860 con 0x7f557013f660 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557009db40 con 0x7f557009c950 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5577216640 10 monclient(hunting): authenticate will time out at 787.646515s 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5574fb2640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f557009c950 0x7f55700a7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56670/0 (socket says 172.21.15.121:56670) 2024-09-13T13:51:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.016+0000 7f5574fb2640 1 -- 172.21.15.121:0/2414689567 learned_addr learned my addr 172.21.15.121:0/2414689567 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556ffff640 1 -- 172.21.15.121:0/2414689567 >> v1:172.21.15.121:6791/0 conn(0x7f5570135b50 legacy=0x7f557013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f5574fb2640 1 -- 172.21.15.121:0/2414689567 >> v1:172.21.15.121:6793/0 conn(0x7f557009c950 legacy=0x7f55700a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870711574 0 0) 0x7f5570135f50 con 0x7f5570135b50 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): my global_id is 4333 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f555c003520 con 0x7f5570135b50 2024-09-13T13:51:48.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118246071 0 0) 0x7f557009db40 con 0x7f557009c950 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 10 monclient(hunting): my global_id is 4400 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.018+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5570135f50 con 0x7f557009c950 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f55757b3640 1 -- 172.21.15.121:0/2414689567 >> v1:172.21.15.153:6791/0 conn(0x7f557013f660 legacy=0x7f5570141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3025241424 0 0) 0x7f555c003520 con 0x7f5570135b50 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 >> v1:172.21.15.121:6793/0 conn(0x7f557009c950 legacy=0x7f55700a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 >> v1:172.21.15.153:6791/0 conn(0x7f557013f660 legacy=0x7f5570141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 monclient: found mon.noname-c 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:48.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557009de20 con 0x7f5570135b50 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5558001b30 con 0x7f5570135b50 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 5 monclient: authenticate success, global_id 4333 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: dump: 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.026 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5558003610 con 0x7f5570135b50 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 1 -- 172.21.15.121:0/2414689567 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5558003b00 con 0x7f5570135b50 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 4 set_mon_vals no callback set 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f556f7fe640 10 monclient: dump: 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.028 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 10 monclient: shutdown 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 1 -- 172.21.15.121:0/2414689567 >> v1:172.21.15.121:6791/0 conn(0x7f5570135b50 legacy=0x7f557013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 1 -- 172.21.15.121:0/2414689567 shutdown_connections 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 1 -- 172.21.15.121:0/2414689567 >> 172.21.15.121:0/2414689567 conn(0x7f55700ab760 msgr2=0x7f55700abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 1 -- 172.21.15.121:0/2414689567 shutdown_connections 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.019+0000 7f5577216640 1 -- 172.21.15.121:0/2414689567 wait complete. 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.020+0000 7f5577216640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.020+0000 7f5577216640 10 monclient: monmap: 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.030 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.020+0000 7f5577216640 1 Processor -- start 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.020+0000 7f5577216640 1 -- start start 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.020+0000 7f5577216640 10 monclient: init 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient: _add_conns ranks=[0,5,7,8,2,4,6,1,3] 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient(hunting): picked mon.noname-a con 0x7f557009c950 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient(hunting): picked mon.noname-h con 0x7f5570135b50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient(hunting): picked mon.noname-i con 0x7f557013f660 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55701352e0 con 0x7f557009c950 2024-09-13T13:51:48.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55700a5690 con 0x7f5570135b50 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557013e0a0 con 0x7f557013f660 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5577216640 10 monclient(hunting): authenticate will time out at 787.651442s 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5574fb2640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f557009c950 0x7f557013a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45564/0 (socket says 172.21.15.121:45564) 2024-09-13T13:51:48.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.021+0000 7f5574fb2640 1 -- 172.21.15.121:0/15471136 learned_addr learned my addr 172.21.15.121:0/15471136 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f5574fb2640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.121:6789/0 conn(0x7f557009c950 legacy=0x7f557013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4171174108 0 0) 0x7f5560001500 con 0x7f557009c950 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): my global_id is 4455 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5550003550 con 0x7f557009c950 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556ffff640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.153:6791/0 conn(0x7f5570135b50 legacy=0x7f557013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f55757b3640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.153:6792/0 conn(0x7f557013f660 legacy=0x7f55701d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3891953346 0 0) 0x7f5550003550 con 0x7f557009c950 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5560001500 con 0x7f557009c950 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5560004b20 con 0x7f557009c950 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.023+0000 7f556d7fa640 10 monclient(hunting): dump: 2024-09-13T13:51:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1123825088 0 0) 0x7f5560001500 con 0x7f557009c950 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.153:6792/0 conn(0x7f557013f660 legacy=0x7f55701d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.153:6791/0 conn(0x7f5570135b50 legacy=0x7f557013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 monclient: found mon.a 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557013e380 con 0x7f557009c950 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 5 monclient: authenticate success, global_id 4455 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 10 monclient: _renew_subs 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f557013c020 con 0x7f557009c950 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 10 monclient: _renew_subs 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f557013ce50 con 0x7f557009c950 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5560004f70 con 0x7f557009c950 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5560005460 con 0x7f557009c950 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556effd640 4 set_mon_vals no callback set 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 10 monclient: dump: 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.038 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.024+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f5560011b60 con 0x7f557009c950 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.025+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7f556004e4e0 con 0x7f557009c950 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.025+0000 7f5577216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.025+0000 7f5577216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.025+0000 7f5577216640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.025+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5554005080 con 0x7f557009c950 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.025+0000 7f556ffff640 10 monclient: get_auth_request con 0x7f555003fae0 auth_method 0 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.029+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f55701352e0 con 0x7f557009c950 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.029+0000 7f556d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.029+0000 7f556d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:48.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.201+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5554000b90 con 0x7f555003fae0 2024-09-13T13:51:48.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.203+0000 7f556d7fa640 1 -- 172.21.15.121:0/15471136 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34163 (unknown 2965378022 0 3406736524) 0x7f5554000b90 con 0x7f555003fae0 2024-09-13T13:51:48.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:51:48.205 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.153:6812/1206688214 conn(0x7f555003fae0 legacy=0x7f5550041fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 10 monclient: shutdown 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 >> v1:172.21.15.121:6789/0 conn(0x7f557009c950 legacy=0x7f557013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 shutdown_connections 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 >> 172.21.15.121:0/15471136 conn(0x7f55700ab760 msgr2=0x7f5570134850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 shutdown_connections 2024-09-13T13:51:48.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.205+0000 7f5577216640 1 -- 172.21.15.121:0/15471136 wait complete. 2024-09-13T13:51:48.217 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-13T13:51:47.219537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163312,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067728,"statfs":{"total":579820584960,"available":579653353472,"internally_reserved":0,"allocated":1146880,"data_stored":1385212,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.771462"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470031+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:51:43.470031+0000","last_peered":"2024-09-13T13:51:43.470031+0000","last_clean":"2024-09-13T13:51:43.470031+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:51:43.470031+0000","last_undegraded":"2024-09-13T13:51:43.470031+0000","last_fullsized":"2024-09-13T13:51:43.470031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.347416+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:51:46.347416+0000","last_peered":"2024-09-13T13:51:46.347416+0000","last_clean":"2024-09-13T13:51:46.347416+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:51:46.347416+0000","last_undegraded":"2024-09-13T13:51:46.347416+0000","last_fullsized":"2024-09-13T13:51:46.347416+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470103+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:51:43.470103+0000","last_peered":"2024-09-13T13:51:43.470103+0000","last_clean":"2024-09-13T13:51:43.470103+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:51:43.470103+0000","last_undegraded":"2024-09-13T13:51:43.470103+0000","last_fullsized":"2024-09-13T13:51:43.470103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-13T13:51:46.347665+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:51:46.347665+0000","last_peered":"2024-09-13T13:51:46.347665+0000","last_clean":"2024-09-13T13:51:46.347665+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:51:46.347665+0000","last_undegraded":"2024-09-13T13:51:46.347665+0000","last_fullsized":"2024-09-13T13:51:46.347665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.472412+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:51:43.472412+0000","last_peered":"2024-09-13T13:51:43.472412+0000","last_clean":"2024-09-13T13:51:43.472412+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:51:43.472412+0000","last_undegraded":"2024-09-13T13:51:43.472412+0000","last_fullsized":"2024-09-13T13:51:43.472412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.467703+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:51:43.467703+0000","last_peered":"2024-09-13T13:51:43.467703+0000","last_clean":"2024-09-13T13:51:43.467703+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:51:43.467703+0000","last_undegraded":"2024-09-13T13:51:43.467703+0000","last_fullsized":"2024-09-13T13:51:43.467703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470775+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:51:43.470775+0000","last_peered":"2024-09-13T13:51:43.470775+0000","last_clean":"2024-09-13T13:51:43.470775+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:51:43.470775+0000","last_undegraded":"2024-09-13T13:51:43.470775+0000","last_fullsized":"2024-09-13T13:51:43.470775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.471333+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:51:43.471333+0000","last_peered":"2024-09-13T13:51:43.471333+0000","last_clean":"2024-09-13T13:51:43.471333+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:51:43.471333+0000","last_undegraded":"2024-09-13T13:51:43.471333+0000","last_fullsized":"2024-09-13T13:51:43.471333+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T13:51:43.470030+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:51:43.470030+0000","last_peered":"2024-09-13T13:51:43.470030+0000","last_clean":"2024-09-13T13:51:43.470030+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:51:43.470030+0000","last_undegraded":"2024-09-13T13:51:43.470030+0000","last_fullsized":"2024-09-13T13:51:43.470030+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":54572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":225280,"data_stored":583481,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:51:48.218 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-13T13:51:48.219 DEBUG:tasks.mon_thrash:Yielding 2024-09-13T13:51:48.219 INFO:teuthology.run_tasks:Running task exec... 2024-09-13T13:51:48.229 INFO:teuthology.task.exec:Executing custom commands... 2024-09-13T13:51:48.230 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:51:48.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_rados_delete_pools_parallel --debug_objecter 20 --debug_ms 1 --debug_rados 20 --debug_monc 20' 2024-09-13T13:51:48.230 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-13T13:51:48.230 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726235508, revive delay: 20.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 20.0 2024-09-13T13:51:48.230 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-13T13:51:48.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T13:51:48.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.327+0000 7f55b6f64640 10 monclient: get_monmap_and_config 2024-09-13T13:51:48.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.327+0000 7f55b6f64640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.327+0000 7f55b6f64640 10 monclient: monmap: 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:48.329368+0000 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:48.329368+0000 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.330 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.331 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.331 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.331 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 1 Processor -- start 2024-09-13T13:51:48.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 1 -- start start 2024-09-13T13:51:48.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient: init 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient: _add_conns ranks=[7,1,3,4,6,8,5,2,0] 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient(hunting): picked mon.noname-i con 0x7f55b009c950 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient(hunting): picked mon.noname-f con 0x7f55b0135b50 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient(hunting): picked mon.noname-g con 0x7f55b013f660 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b0135f50 con 0x7f55b0135b50 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b009d860 con 0x7f55b013f660 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b009db40 con 0x7f55b009c950 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.328+0000 7f55b6f64640 10 monclient(hunting): authenticate will time out at 787.958963s 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.329+0000 7f55b5501640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f55b013f660 0x7f55b0141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:48032/0 (socket says 172.21.15.121:48032) 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.329+0000 7f55b5501640 1 -- 172.21.15.121:0/4264179838 learned_addr learned my addr 172.21.15.121:0/4264179838 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.329+0000 7f55a7fff640 1 --1- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6789/0 conn(0x7f55b0135b50 0x7f55b013df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.121:39122/0 (socket says 172.21.15.121:39122) 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b5501640 1 -- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6790/0 conn(0x7f55b013f660 legacy=0x7f55b0141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a7fff640 1 -- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6789/0 conn(0x7f55b0135b50 legacy=0x7f55b013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b4d00640 1 -- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6792/0 conn(0x7f55b009c950 legacy=0x7f55b00a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238601918 0 0) 0x7f55b009d860 con 0x7f55b013f660 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): my global_id is 4359 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5598003520 con 0x7f55b013f660 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1854267954 0 0) 0x7f55b0135f50 con 0x7f55b0135b50 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): my global_id is 4366 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b009d860 con 0x7f55b0135b50 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 609531418 0 0) 0x7f55b009db40 con 0x7f55b009c950 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient(hunting): my global_id is 4417 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b0135f50 con 0x7f55b009c950 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1003730531 0 0) 0x7f5598003520 con 0x7f55b013f660 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6792/0 conn(0x7f55b009c950 legacy=0x7f55b00a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6789/0 conn(0x7f55b0135b50 legacy=0x7f55b013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 monclient: found mon.noname-g 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b009de20 con 0x7f55b013f660 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f55a8001ae0 con 0x7f55b013f660 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b6f64640 5 monclient: authenticate success, global_id 4359 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b6f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: dump: 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.340 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.341 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b6f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55a80048a0 con 0x7f55b013f660 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 1 -- 172.21.15.121:0/4264179838 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f55a8004e50 con 0x7f55b013f660 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b6f64640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55b6f64640 4 set_mon_vals no callback set 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.331+0000 7f55a77fe640 10 monclient: dump: 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.342 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 10 monclient: shutdown 2024-09-13T13:51:48.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 1 -- 172.21.15.121:0/4264179838 >> v1:172.21.15.153:6790/0 conn(0x7f55b013f660 legacy=0x7f55b0141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 1 -- 172.21.15.121:0/4264179838 shutdown_connections 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 1 -- 172.21.15.121:0/4264179838 >> 172.21.15.121:0/4264179838 conn(0x7f55b00ab760 msgr2=0x7f55b00abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 1 -- 172.21.15.121:0/4264179838 shutdown_connections 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.332+0000 7f55b6f64640 1 -- 172.21.15.121:0/4264179838 wait complete. 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.333+0000 7f55b6f64640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.333+0000 7f55b6f64640 10 monclient: monmap: 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:48.344 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.333+0000 7f55b6f64640 1 Processor -- start 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.333+0000 7f55b6f64640 1 -- start start 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.333+0000 7f55b6f64640 10 monclient: init 2024-09-13T13:51:48.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient: _add_conns ranks=[3,6,1,2,4,8,7,0,5] 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient(hunting): picked mon.noname-g con 0x7f55b009c950 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient(hunting): picked mon.noname-d con 0x7f55b0135b50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient(hunting): picked mon.noname-f con 0x7f55b013f660 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b00a4ee0 con 0x7f55b013f660 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b0142e50 con 0x7f55b009c950 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55b013e400 con 0x7f55b0135b50 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55b6f64640 10 monclient(hunting): authenticate will time out at 787.964176s 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55a7fff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f55b0135b50 0x7f55b013b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52416/0 (socket says 172.21.15.121:52416) 2024-09-13T13:51:48.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.334+0000 7f55a7fff640 1 -- 172.21.15.121:0/643964716 learned_addr learned my addr 172.21.15.121:0/643964716 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a7fff640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.121:6792/0 conn(0x7f55b0135b50 legacy=0x7f55b013b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55b5501640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.153:6789/0 conn(0x7f55b013f660 legacy=0x7f55b013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55b4d00640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.153:6790/0 conn(0x7f55b009c950 legacy=0x7f55b013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768278372 0 0) 0x7f55b013e400 con 0x7f55b0135b50 2024-09-13T13:51:48.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): my global_id is 4398 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f559c003520 con 0x7f55b0135b50 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109556301 0 0) 0x7f55b00a4ee0 con 0x7f55b013f660 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): my global_id is 4375 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b013e400 con 0x7f55b013f660 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577550430 0 0) 0x7f55b0135f50 con 0x7f55b009c950 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): my global_id is 4368 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55b00a4ee0 con 0x7f55b009c950 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1346164366 0 0) 0x7f559c003520 con 0x7f55b0135b50 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55b0135f50 con 0x7f55b0135b50 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f55a00047e0 con 0x7f55b0135b50 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): dump: 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.352 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2340237199 0 0) 0x7f55b013e400 con 0x7f55b013f660 2024-09-13T13:51:48.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f559c003520 con 0x7f55b013f660 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3203582829 0 0) 0x7f55b00a4ee0 con 0x7f55b009c950 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f55b013e400 con 0x7f55b009c950 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f55a8001d30 con 0x7f55b013f660 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): dump: 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.354 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f55ac003990 con 0x7f55b009c950 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient(hunting): dump: 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.355 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2260111 0 0) 0x7f55b0135f50 con 0x7f55b0135b50 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.153:6790/0 conn(0x7f55b009c950 legacy=0x7f55b013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.153:6789/0 conn(0x7f55b013f660 legacy=0x7f55b013b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 1 monclient: found mon.d 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.336+0000 7f55a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b013e6e0 con 0x7f55b0135b50 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 5 monclient: authenticate success, global_id 4398 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 monclient: _renew_subs 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f55b0142a70 con 0x7f55b0135b50 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 monclient: _renew_subs 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f55b013e120 con 0x7f55b0135b50 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f55a0004cb0 con 0x7f55b0135b50 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f55a00051a0 con 0x7f55b0135b50 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a6ffd640 4 set_mon_vals no callback set 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 10 monclient: dump: 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.358 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.359 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f55a0011920 con 0x7f55b0135b50 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.337+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5590005080 con 0x7f55b0135b50 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.338+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (unknown 196223855 0 0) 0x7f55a004e2f0 con 0x7f55b0135b50 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.338+0000 7f55b4d00640 10 monclient: get_auth_request con 0x7f559c040700 auth_method 0 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.342+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f55a0017950 con 0x7f55b0135b50 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.342+0000 7f55a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.342+0000 7f55a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.593+0000 7f55b6f64640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.593+0000 7f55b6f64640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.593+0000 7f55b6f64640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.593+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5590005370 con 0x7f55b0135b50 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.594+0000 7f55a57fa640 1 -- 172.21.15.121:0/643964716 <== mon.6 v1:172.21.15.121:6792/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2745 (unknown 254746657 0 3232299530) 0x7f55a0017250 con 0x7f55b0135b50 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.594+0000 7f55a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.594+0000 7f55a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:48.596 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":14,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","modified":"2024-09-13T13:50:47.391384Z","created":"2024-09-13T13:50:47.391384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6791","nonce":0}]},"addr":"172.21.15.121:6791/0","public_addr":"172.21.15.121:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6791","nonce":0}]},"addr":"172.21.15.153:6791/0","public_addr":"172.21.15.153:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6792","nonce":0}]},"addr":"172.21.15.121:6792/0","public_addr":"172.21.15.121:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6792","nonce":0}]},"addr":"172.21.15.153:6792/0","public_addr":"172.21.15.153:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6793","nonce":0}]},"addr":"172.21.15.121:6793/0","public_addr":"172.21.15.121:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.153:6812/1206688214 conn(0x7f559c040700 legacy=0x7f559c042bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 10 monclient: shutdown 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 >> v1:172.21.15.121:6792/0 conn(0x7f55b0135b50 legacy=0x7f55b013b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 shutdown_connections 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 >> 172.21.15.121:0/643964716 conn(0x7f55b00ab760 msgr2=0x7f55b0134e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 shutdown_connections 2024-09-13T13:51:48.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.596+0000 7f55b6f64640 1 -- 172.21.15.121:0/643964716 wait complete. 2024-09-13T13:51:48.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-13T13:51:48.608 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-13T13:51:48.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-13T13:51:48.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.705+0000 7f2ec90f7640 10 monclient: get_monmap_and_config 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.705+0000 7f2ec90f7640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.705+0000 7f2ec90f7640 10 monclient: monmap: 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:48.706904+0000 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:48.706904+0000 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.707 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.708 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.705+0000 7f2ec90f7640 1 Processor -- start 2024-09-13T13:51:48.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.705+0000 7f2ec90f7640 1 -- start start 2024-09-13T13:51:48.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.705+0000 7f2ec90f7640 10 monclient: init 2024-09-13T13:51:48.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient: _add_conns ranks=[4,2,6,5,8,0,3,7,1] 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec40a6160 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec409d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient(hunting): picked mon.noname-d con 0x7f2ec40a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec409e210 con 0x7f2ec409d150 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec40a7ef0 con 0x7f2ec40a6160 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec409d970 con 0x7f2ec40a6c50 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec90f7640 10 monclient(hunting): authenticate will time out at 788.336330s 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec2d9d640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f2ec40a6160 0x7f2ec40a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33842/0 (socket says 172.21.15.121:33842) 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec259c640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f2ec409d150 0x7f2ec413e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59206/0 (socket says 172.21.15.121:59206) 2024-09-13T13:51:48.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.706+0000 7f2ec2d9d640 1 -- 172.21.15.121:0/705318128 learned_addr learned my addr 172.21.15.121:0/705318128 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec359e640 1 -- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6792/0 conn(0x7f2ec40a6c50 legacy=0x7f2ec4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec259c640 1 -- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6790/0 conn(0x7f2ec409d150 legacy=0x7f2ec413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec2d9d640 1 -- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6791/0 conn(0x7f2ec40a6160 legacy=0x7f2ec40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3717391870 0 0) 0x7f2ec409d970 con 0x7f2ec40a6c50 2024-09-13T13:51:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): my global_id is 4407 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2eac003520 con 0x7f2ec40a6c50 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 425655466 0 0) 0x7f2ec409e210 con 0x7f2ec409d150 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): my global_id is 4376 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec409d970 con 0x7f2ec409d150 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1245010945 0 0) 0x7f2ec40a7ef0 con 0x7f2ec40a6160 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 10 monclient(hunting): my global_id is 4351 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.708+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec409e210 con 0x7f2ec40a6160 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3875063138 0 0) 0x7f2ec409e210 con 0x7f2ec40a6160 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec359e640 1 --1- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6792/0 conn(0x7f2ec40a6c50 0x7f2ec4141aa0 :-1 s=OPENED pgs=77 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7f2ebc001ef0 mon_map magic: 0 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6792/0 conn(0x7f2ec40a6c50 legacy=0x7f2ec4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6790/0 conn(0x7f2ec409d150 legacy=0x7f2ec413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 monclient: found mon.noname-c 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec409e630 con 0x7f2ec40a6160 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2eb4001a00 con 0x7f2ec40a6160 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 5 monclient: authenticate success, global_id 4351 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: dump: 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2ebc001ef0 con 0x7f2ec40a6c50 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2eb4001c50 con 0x7f2ec40a6160 2024-09-13T13:51:48.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 1 -- 172.21.15.121:0/705318128 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2eb4004be0 con 0x7f2ec40a6160 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 4 set_mon_vals no callback set 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec1d9b640 10 monclient: dump: 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 10 monclient: shutdown 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 1 -- 172.21.15.121:0/705318128 >> v1:172.21.15.121:6791/0 conn(0x7f2ec40a6160 legacy=0x7f2ec40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 1 -- 172.21.15.121:0/705318128 shutdown_connections 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 1 -- 172.21.15.121:0/705318128 >> 172.21.15.121:0/705318128 conn(0x7f2ec40ab940 msgr2=0x7f2ec40abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 1 -- 172.21.15.121:0/705318128 shutdown_connections 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.709+0000 7f2ec90f7640 1 -- 172.21.15.121:0/705318128 wait complete. 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.710+0000 7f2ec90f7640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.710+0000 7f2ec90f7640 10 monclient: monmap: 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.710+0000 7f2ec90f7640 1 Processor -- start 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 1 -- start start 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient: init 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient: _add_conns ranks=[4,1,2,8,7,5,3,0,6] 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec409d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient(hunting): picked mon.noname-f con 0x7f2ec40a6160 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec40a6c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec4125a50 con 0x7f2ec40a6160 2024-09-13T13:51:48.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec41cbb50 con 0x7f2ec40a6c50 2024-09-13T13:51:48.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec41263f0 con 0x7f2ec409d150 2024-09-13T13:51:48.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec90f7640 10 monclient(hunting): authenticate will time out at 788.341527s 2024-09-13T13:51:48.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec2d9d640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f2ec409d150 0x7f2ec41392b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33854/0 (socket says 172.21.15.121:33854) 2024-09-13T13:51:48.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.711+0000 7f2ec2d9d640 1 -- 172.21.15.121:0/3964742390 learned_addr learned my addr 172.21.15.121:0/3964742390 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2ec359e640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6790/0 conn(0x7f2ec40a6c50 legacy=0x7f2ec41d6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2ec2d9d640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6791/0 conn(0x7f2ec409d150 legacy=0x7f2ec41392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798555043 0 0) 0x7f2ec409e630 con 0x7f2ec409d150 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): my global_id is 4360 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2eb0003520 con 0x7f2ec409d150 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249608337 0 0) 0x7f2ebc0019b0 con 0x7f2ec40a6c50 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 10 monclient(hunting): my global_id is 4385 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.713+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec409e630 con 0x7f2ec40a6c50 2024-09-13T13:51:48.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec259c640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.153:6789/0 conn(0x7f2ec40a6160 legacy=0x7f2ec41d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3721566134 0 0) 0x7f2eb0003520 con 0x7f2ec409d150 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ebc0019b0 con 0x7f2ec409d150 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2eb4004b20 con 0x7f2ec409d150 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.729 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1197124621 0 0) 0x7f2ec409e630 con 0x7f2ec40a6c50 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2eb0003520 con 0x7f2ec40a6c50 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2ebc003cd0 con 0x7f2ec40a6c50 2024-09-13T13:51:48.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): dump: 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.731 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1769593600 0 0) 0x7f2ec4125a50 con 0x7f2ec40a6160 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient(hunting): my global_id is 4384 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ec409e630 con 0x7f2ec40a6160 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2375901602 0 0) 0x7f2ebc0019b0 con 0x7f2ec409d150 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6790/0 conn(0x7f2ec40a6c50 legacy=0x7f2ec41d6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.153:6789/0 conn(0x7f2ec40a6160 legacy=0x7f2ec41d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 monclient: found mon.c 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec4124930 con 0x7f2ec409d150 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 5 monclient: authenticate success, global_id 4360 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 10 monclient: _renew_subs 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec4126c90 con 0x7f2ec409d150 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 10 monclient: _renew_subs 2024-09-13T13:51:48.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ec4125670 con 0x7f2ec409d150 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2eb4001c50 con 0x7f2ec409d150 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2eb4005490 con 0x7f2ec409d150 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2ec159a640 4 set_mon_vals no callback set 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.714+0000 7f2eab7fe640 10 monclient: dump: 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.734 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.715+0000 7f2ec90f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.715+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ea40065c0 con 0x7f2ea4002860 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.715+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f2eb4011b10 con 0x7f2ec409d150 2024-09-13T13:51:48.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.715+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 3368173399 0 0) 0x7f2eb404e320 con 0x7f2ec409d150 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.715+0000 7f2ec359e640 10 monclient: get_auth_request con 0x7f2eb0040240 auth_method 0 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2ec259c640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6789/0 conn(0x7f2ea4002860 legacy=0x7f2ea4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223253807 0 0) 0x7f2ea40065c0 con 0x7f2ea4002860 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 10 monclient(hunting): my global_id is 4473 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ebc0019b0 con 0x7f2ea4002860 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 462365884 0 0) 0x7f2ebc0019b0 con 0x7f2ea4002860 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ea40065c0 con 0x7f2ea4002860 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2eb80047e0 con 0x7f2ea4002860 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.717+0000 7f2eab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:48.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 10 monclient: dump: 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.737 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1049817526 0 0) 0x7f2ea40065c0 con 0x7f2ea4002860 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ea400a160 con 0x7f2ea4002860 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2eb80049e0 con 0x7f2ea4002860 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6789/0 conn(0x7f2ea4002860 legacy=0x7f2ea4004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.718+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6789/0 conn(0x7f2ea4002860 legacy=0x7f2ea4004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.724+0000 7f2ec90f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:48.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.724+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2ea4009e50 con 0x7f2ea4009110 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.726+0000 7f2ec259c640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6789/0 conn(0x7f2ea4009110 legacy=0x7f2ea40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320943900 0 0) 0x7f2ea4009e50 con 0x7f2ea4009110 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient(hunting): my global_id is 4482 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea40065c0 con 0x7f2ea4009110 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3941606310 0 0) 0x7f2ea40065c0 con 0x7f2ea4009110 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ea4009e50 con 0x7f2ea4009110 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2eb80051d0 con 0x7f2ea4009110 2024-09-13T13:51:48.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient: dump: 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.741 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1173795316 0 0) 0x7f2ea4009e50 con 0x7f2ea4009110 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 --> v1:172.21.15.121:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ea400c3f0 con 0x7f2ea4009110 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 <== mon.0 v1:172.21.15.121:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 3396263618) 0x7f2eb8007410 con 0x7f2ea4009110 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6789/0 conn(0x7f2ea4009110 legacy=0x7f2ea40095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.727+0000 7f2eab7fe640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6789/0 conn(0x7f2ea4009110 legacy=0x7f2ea40095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 32385, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:48.743 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 27, 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:48.744 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:48.745 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:48.746 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:48.747 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.748 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.749 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.750 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:48.752 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.753 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.754 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.755 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:48.756 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.757 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.758 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:48.759 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:48.760 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.761 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.762 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.763 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:48.763 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:48.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.153:6812/1206688214 conn(0x7f2eb0040240 legacy=0x7f2eb0042700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 10 monclient: shutdown 2024-09-13T13:51:48.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 >> v1:172.21.15.121:6791/0 conn(0x7f2ec409d150 legacy=0x7f2ec41392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec359e640 1 -- 172.21.15.121:0/3964742390 reap_dead start 2024-09-13T13:51:48.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec359e640 1 -- 172.21.15.121:0/3964742390 reap_dead start 2024-09-13T13:51:48.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 shutdown_connections 2024-09-13T13:51:48.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 >> 172.21.15.121:0/3964742390 conn(0x7f2ec40ab940 msgr2=0x7f2ec41c1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 shutdown_connections 2024-09-13T13:51:48.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.728+0000 7f2ec90f7640 1 -- 172.21.15.121:0/3964742390 wait complete. 2024-09-13T13:51:48.765 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:48.765 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-13T13:51:48.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T13:51:48.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.862+0000 7f7983bab640 10 monclient: get_monmap_and_config 2024-09-13T13:51:48.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.862+0000 7f7983bab640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.862+0000 7f7983bab640 10 monclient: monmap: 2024-09-13T13:51:48.865 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.865 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:48.864412+0000 2024-09-13T13:51:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:48.864412+0000 2024-09-13T13:51:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.866 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.867 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.867 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.867 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.867 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.867 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.867 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.868 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.868 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 1 Processor -- start 2024-09-13T13:51:48.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 1 -- start start 2024-09-13T13:51:48.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient: init 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient: _add_conns ranks=[5,2,4,7,1,8,3,0,6] 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient(hunting): picked mon.noname-h con 0x7f797c0a6160 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient(hunting): picked mon.noname-b con 0x7f797c09d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient(hunting): picked mon.noname-c con 0x7f797c0a6c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c09e210 con 0x7f797c09d150 2024-09-13T13:51:48.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c0a7ef0 con 0x7f797c0a6c50 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c09d970 con 0x7f797c0a6160 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.863+0000 7f7983bab640 10 monclient(hunting): authenticate will time out at 788.493940s 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.864+0000 7f7981146640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f797c09d150 0x7f797c13e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59224/0 (socket says 172.21.15.121:59224) 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.864+0000 7f7981146640 1 -- 172.21.15.121:0/3904000913 learned_addr learned my addr 172.21.15.121:0/3904000913 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7981146640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.121:6790/0 conn(0x7f797c09d150 legacy=0x7f797c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7982148640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.121:6791/0 conn(0x7f797c0a6c50 legacy=0x7f797c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173991131 0 0) 0x7f797c09e210 con 0x7f797c09d150 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7981947640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.153:6791/0 conn(0x7f797c0a6160 legacy=0x7f797c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): my global_id is 4394 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7968003520 con 0x7f797c09d150 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1190610514 0 0) 0x7f797c0a7ef0 con 0x7f797c0a6c50 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): my global_id is 4369 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f797c09e210 con 0x7f797c0a6c50 2024-09-13T13:51:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166076094 0 0) 0x7f797c09d970 con 0x7f797c0a6160 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient(hunting): my global_id is 4379 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f797c0a7ef0 con 0x7f797c0a6160 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3757414464 0 0) 0x7f797c09e210 con 0x7f797c0a6c50 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7981947640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.153:6791/0 conn(0x7f797c0a6160 legacy=0x7f797c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7981947640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.153:6791/0 conn(0x7f797c0a6160 legacy=0x7f797c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7981947640 1 --1- 172.21.15.121:0/3904000913 >> v1:172.21.15.153:6791/0 conn(0x7f797c0a6160 0x7f797c0a6540 :-1 s=OPENED pgs=75 cs=1 l=1).handle_message read tag failed 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.153:6791/0 conn(0x7f797c0a6160 legacy=0x7f797c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7981947640 1 --1- 172.21.15.121:0/3904000913 >> v1:172.21.15.153:6791/0 conn(0x7f797c0a6160 0x7f797c0a6540 :-1 s=OPENED pgs=75 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.121:6790/0 conn(0x7f797c09d150 legacy=0x7f797c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 monclient: found mon.noname-c 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c09e630 con 0x7f797c0a6c50 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 DispatchQueue::entry inject delay of 0.002000 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7983bab640 5 monclient: authenticate success, global_id 4369 2024-09-13T13:51:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.866+0000 7f7983bab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: ms_handle_reset stray mon v1:172.21.15.153:6791/0 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7974001e90 con 0x7f797c0a6c50 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: dump: 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7974003990 con 0x7f797c0a6c50 2024-09-13T13:51:48.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 1 -- 172.21.15.121:0/3904000913 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7974003f00 con 0x7f797c0a6c50 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7980945640 10 monclient: dump: 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7983bab640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.868+0000 7f7983bab640 4 set_mon_vals no callback set 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 10 monclient: shutdown 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 1 -- 172.21.15.121:0/3904000913 >> v1:172.21.15.121:6791/0 conn(0x7f797c0a6c50 legacy=0x7f797c141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 1 -- 172.21.15.121:0/3904000913 shutdown_connections 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 1 -- 172.21.15.121:0/3904000913 >> 172.21.15.121:0/3904000913 conn(0x7f797c0ab940 msgr2=0x7f797c0abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 1 -- 172.21.15.121:0/3904000913 shutdown_connections 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 1 -- 172.21.15.121:0/3904000913 wait complete. 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 10 monclient: build_initial_monmap 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.869+0000 7f7983bab640 10 monclient: monmap: 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:48.890 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:48.891 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 1 Processor -- start 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 1 -- start start 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient: init 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient: _add_conns ranks=[0,8,7,2,4,1,6,3,5] 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient(hunting): picked mon.noname-a con 0x7f797c09d150 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient(hunting): picked mon.noname-e con 0x7f797c0a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient(hunting): picked mon.noname-i con 0x7f797c0a6c50 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c126570 con 0x7f797c09d150 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c126850 con 0x7f797c0a6c50 2024-09-13T13:51:48.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f797c0a8d60 con 0x7f797c0a6160 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7983bab640 10 monclient(hunting): authenticate will time out at 788.500840s 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7981947640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f797c09d150 0x7f797c139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45596/0 (socket says 172.21.15.121:45596) 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.870+0000 7f7981947640 1 -- 172.21.15.121:0/918761426 learned_addr learned my addr 172.21.15.121:0/918761426 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7981947640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6789/0 conn(0x7f797c09d150 legacy=0x7f797c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7981146640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6793/0 conn(0x7f797c0a6160 legacy=0x7f797c1d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108952343 0 0) 0x7f797c126570 con 0x7f797c09d150 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): my global_id is 4491 2024-09-13T13:51:48.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7960003550 con 0x7f797c09d150 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 385764914 0 0) 0x7f797c0a8d60 con 0x7f797c0a6160 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): my global_id is 4418 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f797c126570 con 0x7f797c0a6160 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7982148640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.153:6792/0 conn(0x7f797c0a6c50 legacy=0x7f797c1d60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 97840068 0 0) 0x7f7960003550 con 0x7f797c09d150 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f797c0a8d60 con 0x7f797c09d150 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2238574214 0 0) 0x7f797c126570 con 0x7f797c0a6160 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7960003550 con 0x7f797c0a6160 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684217766 0 0) 0x7f797c126850 con 0x7f797c0a6c50 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): my global_id is 4426 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f797c126570 con 0x7f797c0a6c50 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7978003610 con 0x7f797c09d150 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): dump: 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.895 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f796c0047e0 con 0x7f797c0a6160 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:48.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient(hunting): dump: 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.897 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 311339151 0 0) 0x7f797c0a8d60 con 0x7f797c09d150 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6793/0 conn(0x7f797c0a6160 legacy=0x7f797c1d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.153:6792/0 conn(0x7f797c0a6c50 legacy=0x7f797c1d60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 monclient: found mon.a 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:48.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797c1ce540 con 0x7f797c09d150 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 5 monclient: authenticate success, global_id 4491 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 10 monclient: _renew_subs 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f797c126290 con 0x7f797c09d150 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 10 monclient: _renew_subs 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f797c1cd680 con 0x7f797c09d150 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7978003cb0 con 0x7f797c09d150 2024-09-13T13:51:48.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f79780041a0 con 0x7f797c09d150 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f7973fff640 4 set_mon_vals no callback set 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.873+0000 7f79727fc640 10 monclient: dump: 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.900 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.874+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f7978004340 con 0x7f797c09d150 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.874+0000 7f7983bab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.874+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79640065c0 con 0x7f7964002860 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.874+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 3368173399 0 0) 0x7f797804e480 con 0x7f797c09d150 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.874+0000 7f7982148640 10 monclient: get_auth_request con 0x7f7960040350 auth_method 0 2024-09-13T13:51:48.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.876+0000 7f7981146640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6790/0 conn(0x7f7964002860 legacy=0x7f7964004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.876+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3946729056 0 0) 0x7f79640065c0 con 0x7f7964002860 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.876+0000 7f79727fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.876+0000 7f79727fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.876+0000 7f79727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 10 monclient(hunting): my global_id is 4403 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f797c0a8d60 con 0x7f7964002860 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 204456473 0 0) 0x7f797c0a8d60 con 0x7f7964002860 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79640065c0 con 0x7f7964002860 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f796c005160 con 0x7f7964002860 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 10 monclient: dump: 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.902 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.903 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2364339891 0 0) 0x7f79640065c0 con 0x7f7964002860 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f796400a160 con 0x7f7964002860 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f796c007360 con 0x7f7964002860 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6790/0 conn(0x7f7964002860 legacy=0x7f7964004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.877+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6790/0 conn(0x7f7964002860 legacy=0x7f7964004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.883+0000 7f7983bab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.883+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7964009e70 con 0x7f7964009130 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.885+0000 7f7981146640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6790/0 conn(0x7f7964009130 legacy=0x7f79640095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479081638 0 0) 0x7f7964009e70 con 0x7f7964009130 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient(hunting): my global_id is 4412 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79640065c0 con 0x7f7964009130 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3797066915 0 0) 0x7f79640065c0 con 0x7f7964009130 2024-09-13T13:51:48.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7964009e70 con 0x7f7964009130 2024-09-13T13:51:48.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f796c006b90 con 0x7f7964009130 2024-09-13T13:51:48.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:48.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:48.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient: dump: 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:48.908 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:48.909 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1691006704 0 0) 0x7f7964009e70 con 0x7f7964009130 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 --> v1:172.21.15.121:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f796400c3f0 con 0x7f7964009130 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 <== mon.2 v1:172.21.15.121:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7728 (unknown 0 0 1610859989) 0x7f796c006f20 con 0x7f7964009130 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:48.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:48.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6790/0 conn(0x7f7964009130 legacy=0x7f79640095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.886+0000 7f79727fc640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6790/0 conn(0x7f7964009130 legacy=0x7f79640095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:48.911 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:48.911 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:48.911 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 32554, 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:48.912 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:48.913 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 27, 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:48.914 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:48.915 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:48.916 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:48.917 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:48.918 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:48.919 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:48.920 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:48.921 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:48.922 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:48.923 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.925 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.926 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.927 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.933 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.936 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:48.937 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:48.938 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:48.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:48.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:48.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:48.939 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:48.939 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.153:6812/1206688214 conn(0x7f7960040350 legacy=0x7f7960042810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 10 monclient: shutdown 2024-09-13T13:51:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:48.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 >> v1:172.21.15.121:6789/0 conn(0x7f797c09d150 legacy=0x7f797c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7982148640 1 -- 172.21.15.121:0/918761426 reap_dead start 2024-09-13T13:51:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7982148640 1 -- 172.21.15.121:0/918761426 reap_dead start 2024-09-13T13:51:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 shutdown_connections 2024-09-13T13:51:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 >> 172.21.15.121:0/918761426 conn(0x7f797c0ab940 msgr2=0x7f797c0aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 shutdown_connections 2024-09-13T13:51:48.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:48.887+0000 7f7983bab640 1 -- 172.21.15.121:0/918761426 wait complete. 2024-09-13T13:51:48.941 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:48.941 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-13T13:51:48.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.038+0000 7f2a7c840640 10 monclient: get_monmap_and_config 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.038+0000 7f2a7c840640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.038+0000 7f2a7c840640 10 monclient: monmap: 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:49.039823+0000 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:49.039823+0000 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.040 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.041 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.038+0000 7f2a7c840640 1 Processor -- start 2024-09-13T13:51:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.038+0000 7f2a7c840640 1 -- start start 2024-09-13T13:51:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.038+0000 7f2a7c840640 10 monclient: init 2024-09-13T13:51:49.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient: _add_conns ranks=[7,1,8,6,5,4,0,3,2] 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient(hunting): picked mon.noname-i con 0x7f2a780a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient(hunting): picked mon.noname-f con 0x7f2a7809d150 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient(hunting): picked mon.noname-e con 0x7f2a780a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7809e210 con 0x7f2a7809d150 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a780a7ef0 con 0x7f2a780a6160 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a7809d970 con 0x7f2a780a6c50 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a7c840640 10 monclient(hunting): authenticate will time out at 788.669388s 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a76d9d640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f2a780a6c50 0x7f2a78141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56688/0 (socket says 172.21.15.121:56688) 2024-09-13T13:51:49.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.039+0000 7f2a76d9d640 1 -- 172.21.15.121:0/2717341728 learned_addr learned my addr 172.21.15.121:0/2717341728 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a76d9d640 1 -- 172.21.15.121:0/2717341728 >> v1:172.21.15.121:6793/0 conn(0x7f2a780a6c50 legacy=0x7f2a78141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3766985362 0 0) 0x7f2a7809d970 con 0x7f2a780a6c50 2024-09-13T13:51:49.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7559a640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7559a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7559a640 10 monclient(hunting): my global_id is 4427 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7659c640 1 -- 172.21.15.121:0/2717341728 >> v1:172.21.15.153:6792/0 conn(0x7f2a780a6160 legacy=0x7f2a780a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a60003520 con 0x7f2a780a6c50 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.041+0000 7f2a75d9b640 1 -- 172.21.15.121:0/2717341728 >> v1:172.21.15.153:6789/0 conn(0x7f2a7809d150 legacy=0x7f2a7813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2722175780 0 0) 0x7f2a60003520 con 0x7f2a780a6c50 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 >> v1:172.21.15.153:6792/0 conn(0x7f2a780a6160 legacy=0x7f2a780a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 >> v1:172.21.15.153:6789/0 conn(0x7f2a7809d150 legacy=0x7f2a7813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 monclient: found mon.noname-e 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7809e630 con 0x7f2a780a6c50 2024-09-13T13:51:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 5 monclient: authenticate success, global_id 4427 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a68001ae0 con 0x7f2a780a6c50 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: dump: 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a680048a0 con 0x7f2a780a6c50 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 1 -- 172.21.15.121:0/2717341728 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a68004dd0 con 0x7f2a780a6c50 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 4 set_mon_vals no callback set 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7559a640 10 monclient: dump: 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.051 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 10 monclient: shutdown 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 1 -- 172.21.15.121:0/2717341728 >> v1:172.21.15.121:6793/0 conn(0x7f2a780a6c50 legacy=0x7f2a78141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 1 -- 172.21.15.121:0/2717341728 shutdown_connections 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 1 -- 172.21.15.121:0/2717341728 >> 172.21.15.121:0/2717341728 conn(0x7f2a780ab940 msgr2=0x7f2a780abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 1 -- 172.21.15.121:0/2717341728 shutdown_connections 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.042+0000 7f2a7c840640 1 -- 172.21.15.121:0/2717341728 wait complete. 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.043+0000 7f2a7c840640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.043+0000 7f2a7c840640 10 monclient: monmap: 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.043+0000 7f2a7c840640 1 Processor -- start 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.043+0000 7f2a7c840640 1 -- start start 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.043+0000 7f2a7c840640 10 monclient: init 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient: _add_conns ranks=[3,1,6,4,5,0,2,7,8] 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient(hunting): picked mon.noname-g con 0x7f2a7809d150 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient(hunting): picked mon.noname-f con 0x7f2a780a6160 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient(hunting): picked mon.noname-d con 0x7f2a780a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a78124b30 con 0x7f2a780a6160 2024-09-13T13:51:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a780a8ba0 con 0x7f2a7809d150 2024-09-13T13:51:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a780a4610 con 0x7f2a780a6c50 2024-09-13T13:51:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a7c840640 10 monclient(hunting): authenticate will time out at 788.674473s 2024-09-13T13:51:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a76d9d640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f2a780a6c50 0x7f2a781d64f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52444/0 (socket says 172.21.15.121:52444) 2024-09-13T13:51:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.044+0000 7f2a76d9d640 1 -- 172.21.15.121:0/3361603282 learned_addr learned my addr 172.21.15.121:0/3361603282 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a76d9d640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6792/0 conn(0x7f2a780a6c50 legacy=0x7f2a781d64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a7659c640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.153:6790/0 conn(0x7f2a7809d150 legacy=0x7f2a781394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a75d9b640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.153:6789/0 conn(0x7f2a780a6160 legacy=0x7f2a781d2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345023331 0 0) 0x7f2a780a4610 con 0x7f2a780a6c50 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): my global_id is 4416 2024-09-13T13:51:49.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a58003520 con 0x7f2a780a6c50 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3655836610 0 0) 0x7f2a780a8ba0 con 0x7f2a7809d150 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 10 monclient(hunting): my global_id is 4377 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a780a4610 con 0x7f2a7809d150 2024-09-13T13:51:49.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.046+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370695408 0 0) 0x7f2a78124b30 con 0x7f2a780a6160 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): my global_id is 4402 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a780a8ba0 con 0x7f2a780a6160 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1416480588 0 0) 0x7f2a58003520 con 0x7f2a780a6c50 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a78124b30 con 0x7f2a780a6c50 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a68001d30 con 0x7f2a780a6c50 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.062 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.063 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4119818257 0 0) 0x7f2a780a4610 con 0x7f2a7809d150 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a58003520 con 0x7f2a7809d150 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a70003990 con 0x7f2a7809d150 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient(hunting): dump: 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.064 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1987974444 0 0) 0x7f2a780a8ba0 con 0x7f2a780a6160 2024-09-13T13:51:49.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a780a4610 con 0x7f2a780a6160 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2786757409 0 0) 0x7f2a78124b30 con 0x7f2a780a6c50 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.153:6790/0 conn(0x7f2a7809d150 legacy=0x7f2a781394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.153:6789/0 conn(0x7f2a780a6160 legacy=0x7f2a781d2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 monclient: found mon.d 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a78126e90 con 0x7f2a780a6c50 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 5 monclient: authenticate success, global_id 4416 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 10 monclient: _renew_subs 2024-09-13T13:51:49.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2a781265f0 con 0x7f2a780a6c50 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 10 monclient: _renew_subs 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f2a78139dc0 con 0x7f2a780a6c50 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a680048a0 con 0x7f2a780a6c50 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a68001b00 con 0x7f2a780a6c50 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a74d99640 4 set_mon_vals no callback set 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.047+0000 7f2a66ffd640 10 monclient: dump: 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.067 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.048+0000 7f2a7c840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.048+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2a5c0065c0 con 0x7f2a5c002860 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.048+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f2a68011860 con 0x7f2a780a6c50 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.048+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 3368173399 0 0) 0x7f2a6804e350 con 0x7f2a780a6c50 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.048+0000 7f2a75d9b640 10 monclient: get_auth_request con 0x7f2a58040510 auth_method 0 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a7659c640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6791/0 conn(0x7f2a5c002860 legacy=0x7f2a5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573292903 0 0) 0x7f2a5c0065c0 con 0x7f2a5c002860 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient(hunting): my global_id is 4378 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a78124b30 con 0x7f2a5c002860 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4186847921 0 0) 0x7f2a78124b30 con 0x7f2a5c002860 2024-09-13T13:51:49.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a5c0065c0 con 0x7f2a5c002860 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a700041d0 con 0x7f2a5c002860 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 10 monclient: dump: 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.070 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1056282005 0 0) 0x7f2a5c0065c0 con 0x7f2a5c002860 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.050+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a5c00a160 con 0x7f2a5c002860 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.051+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2a70005b40 con 0x7f2a5c002860 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.051+0000 7f2a66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.051+0000 7f2a66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.051+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6791/0 conn(0x7f2a5c002860 legacy=0x7f2a5c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.051+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6791/0 conn(0x7f2a5c002860 legacy=0x7f2a5c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.057+0000 7f2a7c840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:49.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.057+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2a5c009e70 con 0x7f2a5c009130 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a7659c640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6791/0 conn(0x7f2a5c009130 legacy=0x7f2a5c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412308208 0 0) 0x7f2a5c009e70 con 0x7f2a5c009130 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient(hunting): my global_id is 4387 2024-09-13T13:51:49.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a5c0065c0 con 0x7f2a5c009130 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 520058112 0 0) 0x7f2a5c0065c0 con 0x7f2a5c009130 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a5c009e70 con 0x7f2a5c009130 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2a700046f0 con 0x7f2a5c009130 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.059+0000 7f2a66ffd640 10 monclient: dump: 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.074 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3576964504 0 0) 0x7f2a5c009e70 con 0x7f2a5c009130 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 --> v1:172.21.15.121:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a5c00c410 con 0x7f2a5c009130 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 <== mon.4 v1:172.21.15.121:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7559 (unknown 0 0 3132753114) 0x7f2a70006110 con 0x7f2a5c009130 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:49.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6791/0 conn(0x7f2a5c009130 legacy=0x7f2a5c0095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a66ffd640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6791/0 conn(0x7f2a5c009130 legacy=0x7f2a5c0095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 32729, 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:49.076 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 27, 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.077 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.078 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:49.079 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.080 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:49.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.082 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:49.083 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.084 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.086 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.087 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.088 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.089 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.090 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.092 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:49.093 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.095 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:49.095 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.153:6812/1206688214 conn(0x7f2a58040510 legacy=0x7f2a580429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 10 monclient: shutdown 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 >> v1:172.21.15.121:6792/0 conn(0x7f2a780a6c50 legacy=0x7f2a781d64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a76d9d640 1 -- 172.21.15.121:0/3361603282 reap_dead start 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a76d9d640 1 -- 172.21.15.121:0/3361603282 reap_dead start 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 shutdown_connections 2024-09-13T13:51:49.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 >> 172.21.15.121:0/3361603282 conn(0x7f2a780ab940 msgr2=0x7f2a7809eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 shutdown_connections 2024-09-13T13:51:49.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.060+0000 7f2a7c840640 1 -- 172.21.15.121:0/3361603282 wait complete. 2024-09-13T13:51:49.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:49.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-13T13:51:49.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-13T13:51:49.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.194+0000 7f6e8511e640 10 monclient: get_monmap_and_config 2024-09-13T13:51:49.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.194+0000 7f6e8511e640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.194+0000 7f6e8511e640 10 monclient: monmap: 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:49.196480+0000 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:49.196480+0000 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.197 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.198 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.198 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.198 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.195+0000 7f6e8511e640 1 Processor -- start 2024-09-13T13:51:49.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.195+0000 7f6e8511e640 1 -- start start 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.195+0000 7f6e8511e640 10 monclient: init 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.195+0000 7f6e8511e640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.195+0000 7f6e8511e640 10 monclient: _add_conns ranks=[6,3,5,8,0,7,4,2,1] 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.195+0000 7f6e8511e640 10 monclient(hunting): picked mon.noname-d con 0x7f6e80138bd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 10 monclient(hunting): picked mon.noname-g con 0x7f6e80138fb0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 10 monclient(hunting): picked mon.noname-h con 0x7f6e80139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8009e100 con 0x7f6e80138fb0 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8009db40 con 0x7f6e80139420 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8009de20 con 0x7f6e80138bd0 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e8511e640 10 monclient(hunting): authenticate will time out at 788.826106s 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e7ed9d640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f6e80138bd0 0x7f6e800a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52460/0 (socket says 172.21.15.121:52460) 2024-09-13T13:51:49.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.196+0000 7f6e7ed9d640 1 -- 172.21.15.121:0/631751658 learned_addr learned my addr 172.21.15.121:0/631751658 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7ed9d640 1 -- 172.21.15.121:0/631751658 >> v1:172.21.15.121:6792/0 conn(0x7f6e80138bd0 legacy=0x7f6e800a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e77fff640 1 -- 172.21.15.121:0/631751658 >> v1:172.21.15.153:6790/0 conn(0x7f6e80138fb0 legacy=0x7f6e8013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7f59e640 1 -- 172.21.15.121:0/631751658 >> v1:172.21.15.153:6791/0 conn(0x7f6e80139420 legacy=0x7f6e80141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033002779 0 0) 0x7f6e8009de20 con 0x7f6e80138bd0 2024-09-13T13:51:49.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): my global_id is 4425 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e6c003520 con 0x7f6e80138bd0 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119524279 0 0) 0x7f6e8009e100 con 0x7f6e80138fb0 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): my global_id is 4386 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e8009de20 con 0x7f6e80138fb0 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923635416 0 0) 0x7f6e8009db40 con 0x7f6e80139420 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient(hunting): my global_id is 4388 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e8009e100 con 0x7f6e80139420 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1960630505 0 0) 0x7f6e6c003520 con 0x7f6e80138bd0 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 >> v1:172.21.15.153:6791/0 conn(0x7f6e80139420 legacy=0x7f6e80141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 >> v1:172.21.15.153:6790/0 conn(0x7f6e80138fb0 legacy=0x7f6e8013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 monclient: found mon.noname-d 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e800a5da0 con 0x7f6e80138bd0 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e70001ae0 con 0x7f6e80138bd0 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e8511e640 5 monclient: authenticate success, global_id 4425 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e8511e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.198+0000 7f6e7e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 10 monclient: dump: 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.207 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e700048a0 con 0x7f6e80138bd0 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 1 -- 172.21.15.121:0/631751658 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e70004e10 con 0x7f6e80138bd0 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 4 set_mon_vals no callback set 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e7e59c640 10 monclient: dump: 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.209 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 10 monclient: shutdown 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 1 -- 172.21.15.121:0/631751658 >> v1:172.21.15.121:6792/0 conn(0x7f6e80138bd0 legacy=0x7f6e800a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 1 -- 172.21.15.121:0/631751658 shutdown_connections 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 1 -- 172.21.15.121:0/631751658 >> 172.21.15.121:0/631751658 conn(0x7f6e800ab800 msgr2=0x7f6e800abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 1 -- 172.21.15.121:0/631751658 shutdown_connections 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.199+0000 7f6e8511e640 1 -- 172.21.15.121:0/631751658 wait complete. 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.200+0000 7f6e8511e640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.200+0000 7f6e8511e640 10 monclient: monmap: 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:49.210 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.200+0000 7f6e8511e640 1 Processor -- start 2024-09-13T13:51:49.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.200+0000 7f6e8511e640 1 -- start start 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.200+0000 7f6e8511e640 10 monclient: init 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient: _add_conns ranks=[8,3,6,5,7,4,2,1,0] 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient(hunting): picked mon.noname-e con 0x7f6e80138bd0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient(hunting): picked mon.noname-g con 0x7f6e80138fb0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient(hunting): picked mon.noname-d con 0x7f6e80139420 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e800a76c0 con 0x7f6e80138fb0 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e800a59d0 con 0x7f6e80139420 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e8013b000 con 0x7f6e80138bd0 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e8511e640 10 monclient(hunting): authenticate will time out at 788.831275s 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e7ed9d640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f6e80138bd0 0x7f6e800a3700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56694/0 (socket says 172.21.15.121:56694) 2024-09-13T13:51:49.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.201+0000 7f6e7ed9d640 1 -- 172.21.15.121:0/2371844612 learned_addr learned my addr 172.21.15.121:0/2371844612 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e7ed9d640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6793/0 conn(0x7f6e80138bd0 legacy=0x7f6e800a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e7f59e640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e80139420 legacy=0x7f6e801d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e77fff640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.153:6790/0 conn(0x7f6e80138fb0 legacy=0x7f6e800a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1833355942 0 0) 0x7f6e8013b000 con 0x7f6e80138bd0 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): my global_id is 4436 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e60003520 con 0x7f6e80138bd0 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225552846 0 0) 0x7f6e8009e100 con 0x7f6e80139420 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): my global_id is 4434 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e8013b000 con 0x7f6e80139420 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1911495580 0 0) 0x7f6e60003520 con 0x7f6e80138bd0 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e8009e100 con 0x7f6e80138bd0 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e70004d90 con 0x7f6e80138bd0 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): dump: 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.218 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082050627 0 0) 0x7f6e8009de20 con 0x7f6e80138fb0 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): my global_id is 4395 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e60003520 con 0x7f6e80138fb0 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2482711854 0 0) 0x7f6e8013b000 con 0x7f6e80139420 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e8009de20 con 0x7f6e80139420 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e78002070 con 0x7f6e80139420 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 10 monclient(hunting): dump: 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.220 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.203+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1180148241 0 0) 0x7f6e8009e100 con 0x7f6e80138bd0 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e80139420 legacy=0x7f6e801d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.153:6790/0 conn(0x7f6e80138fb0 legacy=0x7f6e800a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 monclient: found mon.e 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8013b2e0 con 0x7f6e80138bd0 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 5 monclient: authenticate success, global_id 4436 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 10 monclient: _renew_subs 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:49.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6e80139dc0 con 0x7f6e80138bd0 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 10 monclient: _renew_subs 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f6e800a4700 con 0x7f6e80138bd0 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6e70001d30 con 0x7f6e80138bd0 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e70001b00 con 0x7f6e80138bd0 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e7dd9b640 4 set_mon_vals no callback set 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 10 monclient: dump: 2024-09-13T13:51:49.223 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.224 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6e640065c0 con 0x7f6e64002860 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.204+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f6e70011c20 con 0x7f6e80138bd0 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.205+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(22..22 src has 1..22) ==== 3898+0+0 (unknown 3368173399 0 0) 0x7f6e7004e3a0 con 0x7f6e80138bd0 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.205+0000 7f6e7f59e640 10 monclient: get_auth_request con 0x7f6e60040350 auth_method 0 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e77fff640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e64002860 legacy=0x7f6e64004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2139923142 0 0) 0x7f6e640065c0 con 0x7f6e64002860 2024-09-13T13:51:49.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient(hunting): my global_id is 4443 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e8009e100 con 0x7f6e64002860 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2545554783 0 0) 0x7f6e8009e100 con 0x7f6e64002860 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e640065c0 con 0x7f6e64002860 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e68001ca0 con 0x7f6e64002860 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient: dump: 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.227 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2467277570 0 0) 0x7f6e640065c0 con 0x7f6e64002860 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e6400a160 con 0x7f6e64002860 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6e68006890 con 0x7f6e64002860 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:49.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e64002860 legacy=0x7f6e64004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.207+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e64002860 legacy=0x7f6e64004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.213+0000 7f6e8511e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:49.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.213+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6e64009e30 con 0x7f6e64009110 2024-09-13T13:51:49.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e77fff640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e64009110 legacy=0x7f6e64009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298940929 0 0) 0x7f6e64009e30 con 0x7f6e64009110 2024-09-13T13:51:49.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient(hunting): my global_id is 4452 2024-09-13T13:51:49.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e640065c0 con 0x7f6e64009110 2024-09-13T13:51:49.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2741470936 0 0) 0x7f6e640065c0 con 0x7f6e64009110 2024-09-13T13:51:49.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e64009e30 con 0x7f6e64009110 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6e68003d00 con 0x7f6e64009110 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 10 monclient: dump: 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.232 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2629353190 0 0) 0x7f6e64009e30 con 0x7f6e64009110 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.216+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 --> v1:172.21.15.121:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e6400c3f0 con 0x7f6e64009110 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 <== mon.6 v1:172.21.15.121:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7559 (unknown 0 0 417708035) 0x7f6e68006b00 con 0x7f6e64009110 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e777fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:49.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e777fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e64009110 legacy=0x7f6e64009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e777fe640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6792/0 conn(0x7f6e64009110 legacy=0x7f6e64009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 32889, 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:49.234 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 27, 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.235 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.236 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.237 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:49.238 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.239 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.240 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.242 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.244 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.245 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.246 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.249 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.250 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-13T13:51:49.253 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.254 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:49.254 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:49.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.153:6812/1206688214 conn(0x7f6e60040350 legacy=0x7f6e60042810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 10 monclient: shutdown 2024-09-13T13:51:49.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 >> v1:172.21.15.121:6793/0 conn(0x7f6e80138bd0 legacy=0x7f6e800a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e7f59e640 1 -- 172.21.15.121:0/2371844612 reap_dead start 2024-09-13T13:51:49.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e7f59e640 1 -- 172.21.15.121:0/2371844612 reap_dead start 2024-09-13T13:51:49.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 shutdown_connections 2024-09-13T13:51:49.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 >> 172.21.15.121:0/2371844612 conn(0x7f6e800ab800 msgr2=0x7f6e801367a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 shutdown_connections 2024-09-13T13:51:49.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.217+0000 7f6e8511e640 1 -- 172.21.15.121:0/2371844612 wait complete. 2024-09-13T13:51:49.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:49.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-13T13:51:49.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-13T13:51:49.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.354+0000 7f1a404fe640 10 monclient: get_monmap_and_config 2024-09-13T13:51:49.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.354+0000 7f1a404fe640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.354+0000 7f1a404fe640 10 monclient: monmap: 2024-09-13T13:51:49.356 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.356 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.356 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:49.356149+0000 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:49.356149+0000 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.357 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.358 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.355+0000 7f1a404fe640 1 Processor -- start 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.355+0000 7f1a404fe640 1 -- start start 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.355+0000 7f1a404fe640 10 monclient: init 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient: _add_conns ranks=[7,3,2,6,4,5,1,0,8] 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient(hunting): picked mon.noname-i con 0x7f1a380a3600 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:49.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient(hunting): picked mon.noname-g con 0x7f1a380a40f0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient(hunting): picked mon.noname-b con 0x7f1a3809f950 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3809d020 con 0x7f1a3809f950 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3809d4a0 con 0x7f1a380a40f0 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3809d220 con 0x7f1a380a3600 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a3e29a640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7f1a380a3600 0x7f1a380a39e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.121:60854/0 (socket says 172.21.15.121:60854) 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a3e29a640 1 -- 172.21.15.121:0/3212109696 learned_addr learned my addr 172.21.15.121:0/3212109696 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a3da99640 1 --1- 172.21.15.121:0/3212109696 >> v1:172.21.15.153:6790/0 conn(0x7f1a380a40f0 0x7f1a3813a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:48074/0 (socket says 172.21.15.121:48074) 2024-09-13T13:51:49.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.356+0000 7f1a404fe640 10 monclient(hunting): authenticate will time out at 788.986643s 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.358+0000 7f1a3e29a640 1 -- 172.21.15.121:0/3212109696 >> v1:172.21.15.153:6792/0 conn(0x7f1a380a3600 legacy=0x7f1a380a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.358+0000 7f1a3da99640 1 -- 172.21.15.121:0/3212109696 >> v1:172.21.15.153:6790/0 conn(0x7f1a380a40f0 legacy=0x7f1a3813a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.358+0000 7f1a3ea9b640 1 -- 172.21.15.121:0/3212109696 >> v1:172.21.15.121:6790/0 conn(0x7f1a3809f950 legacy=0x7f1a3813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696001238 0 0) 0x7f1a3809d220 con 0x7f1a380a3600 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): my global_id is 4444 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a24003520 con 0x7f1a380a3600 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561578438 0 0) 0x7f1a3809d020 con 0x7f1a3809f950 2024-09-13T13:51:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient(hunting): my global_id is 4421 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a3809d220 con 0x7f1a3809f950 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3843374827 0 0) 0x7f1a24003520 con 0x7f1a380a3600 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 >> v1:172.21.15.153:6790/0 conn(0x7f1a380a40f0 legacy=0x7f1a3813a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 >> v1:172.21.15.121:6790/0 conn(0x7f1a3809f950 legacy=0x7f1a3813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 monclient: found mon.noname-i 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: _send_mon_message to mon.noname-i at v1:172.21.15.153:6792/0 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 --> v1:172.21.15.153:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a380a5100 con 0x7f1a380a3600 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1a34001ae0 con 0x7f1a380a3600 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: dump: 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a404fe640 5 monclient: authenticate success, global_id 4444 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a404fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 <== mon.7 v1:172.21.15.153:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a340048a0 con 0x7f1a380a3600 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 1 -- 172.21.15.121:0/3212109696 <== mon.7 v1:172.21.15.153:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1a34004e10 con 0x7f1a380a3600 2024-09-13T13:51:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a3d298640 10 monclient: dump: 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a404fe640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.359+0000 7f1a404fe640 4 set_mon_vals no callback set 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 10 monclient: shutdown 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 1 -- 172.21.15.121:0/3212109696 >> v1:172.21.15.153:6792/0 conn(0x7f1a380a3600 legacy=0x7f1a380a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 1 -- 172.21.15.121:0/3212109696 shutdown_connections 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 1 -- 172.21.15.121:0/3212109696 >> 172.21.15.121:0/3212109696 conn(0x7f1a3806cee0 msgr2=0x7f1a380a46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 1 -- 172.21.15.121:0/3212109696 shutdown_connections 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.360+0000 7f1a404fe640 1 -- 172.21.15.121:0/3212109696 wait complete. 2024-09-13T13:51:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 10 monclient: monmap: 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 1 Processor -- start 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 1 -- start start 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 10 monclient: init 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.361+0000 7f1a404fe640 10 monclient: _add_conns ranks=[3,6,8,5,0,4,7,1,2] 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 10 monclient(hunting): picked mon.noname-g con 0x7f1a3809f950 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 10 monclient(hunting): picked mon.noname-d con 0x7f1a380a3600 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 10 monclient(hunting): picked mon.noname-e con 0x7f1a380a40f0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a381cc9b0 con 0x7f1a3809f950 2024-09-13T13:51:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a380a6ea0 con 0x7f1a380a3600 2024-09-13T13:51:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a380a1690 con 0x7f1a380a40f0 2024-09-13T13:51:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a404fe640 10 monclient(hunting): authenticate will time out at 788.992089s 2024-09-13T13:51:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a3ea9b640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f1a380a40f0 0x7f1a381d41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56704/0 (socket says 172.21.15.121:56704) 2024-09-13T13:51:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a3da99640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f1a380a3600 0x7f1a381cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52490/0 (socket says 172.21.15.121:52490) 2024-09-13T13:51:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.362+0000 7f1a3ea9b640 1 -- 172.21.15.121:0/3087631376 learned_addr learned my addr 172.21.15.121:0/3087631376 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.365+0000 7f1a3ea9b640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a380a40f0 legacy=0x7f1a381d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.365+0000 7f1a3e29a640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.153:6790/0 conn(0x7f1a3809f950 legacy=0x7f1a381cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a3da99640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6792/0 conn(0x7f1a380a3600 legacy=0x7f1a381cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1032598621 0 0) 0x7f1a381cc9b0 con 0x7f1a3809f950 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): my global_id is 4404 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a1c003520 con 0x7f1a3809f950 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68673645 0 0) 0x7f1a380a6ea0 con 0x7f1a380a3600 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 10 monclient(hunting): my global_id is 4461 2024-09-13T13:51:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.366+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a381cc9b0 con 0x7f1a380a3600 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 887169887 0 0) 0x7f1a381cc9b0 con 0x7f1a380a3600 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a380a6ea0 con 0x7f1a380a3600 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1a28001c60 con 0x7f1a380a3600 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient(hunting): dump: 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3438757085 0 0) 0x7f1a380a6ea0 con 0x7f1a380a3600 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a380a40f0 legacy=0x7f1a381d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.153:6790/0 conn(0x7f1a3809f950 legacy=0x7f1a381cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 monclient: found mon.d 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a381411a0 con 0x7f1a380a3600 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a28004c10 con 0x7f1a380a3600 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1a28005140 con 0x7f1a380a3600 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.367+0000 7f1a2effd640 10 monclient: dump: 2024-09-13T13:51:49.379 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.380 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.368+0000 7f1a404fe640 5 monclient: authenticate success, global_id 4461 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.368+0000 7f1a404fe640 10 monclient: _renew_subs 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.368+0000 7f1a404fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.368+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a38126660 con 0x7f1a380a3600 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.369+0000 7f1a3ca97640 4 set_mon_vals no callback set 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.369+0000 7f1a404fe640 10 monclient: _renew_subs 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.369+0000 7f1a404fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.369+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a38125510 con 0x7f1a380a3600 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.371+0000 7f1a404fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.371+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1a380766c0 con 0x7f1a3811e240 2024-09-13T13:51:49.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.372+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1a280052e0 con 0x7f1a380a3600 2024-09-13T13:51:49.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.372+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(23..23 src has 1..23) ==== 3898+0+0 (unknown 566398313 0 0) 0x7f1a2804e2a0 con 0x7f1a380a3600 2024-09-13T13:51:49.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.374+0000 7f1a3e29a640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a3811e240 legacy=0x7f1a3811e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a3ea9b640 10 monclient: get_auth_request con 0x7f1a1c03fcd0 auth_method 0 2024-09-13T13:51:49.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132139529 0 0) 0x7f1a1c003520 con 0x7f1a3811e240 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a2effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a2effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a2effd640 10 monclient(hunting): my global_id is 4454 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.375+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a380a6ea0 con 0x7f1a3811e240 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2168144660 0 0) 0x7f1a380a6ea0 con 0x7f1a3811e240 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a1c003520 con 0x7f1a3811e240 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1a34001d30 con 0x7f1a3811e240 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 10 monclient: dump: 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.384 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1600692176 0 0) 0x7f1a1c003520 con 0x7f1a3811e240 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.376+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a38069dc0 con 0x7f1a3811e240 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.377+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1a34004ad0 con 0x7f1a3811e240 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.377+0000 7f1a2effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.377+0000 7f1a2effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.377+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a3811e240 legacy=0x7f1a3811e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.377+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a3811e240 legacy=0x7f1a3811e620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.385+0000 7f1a404fe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:49.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.385+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1a3806dcd0 con 0x7f1a3811ed30 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a3e29a640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a3811ed30 legacy=0x7f1a381d98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720641050 0 0) 0x7f1a3806dcd0 con 0x7f1a3811ed30 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient(hunting): my global_id is 4463 2024-09-13T13:51:49.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a1c003520 con 0x7f1a3811ed30 2024-09-13T13:51:49.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1547211623 0 0) 0x7f1a1c003520 con 0x7f1a3811ed30 2024-09-13T13:51:49.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a3806dcd0 con 0x7f1a3811ed30 2024-09-13T13:51:49.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1a34005490 con 0x7f1a3811ed30 2024-09-13T13:51:49.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:49.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.387+0000 7f1a2effd640 10 monclient: dump: 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.392 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.393 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2305663999 0 0) 0x7f1a3806dcd0 con 0x7f1a3811ed30 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 --> v1:172.21.15.121:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1a3811f110 con 0x7f1a3811ed30 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 <== mon.8 v1:172.21.15.121:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7559 (unknown 0 0 2623876488) 0x7f1a34008600 con 0x7f1a3811ed30 2024-09-13T13:51:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:49.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:49.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a3811ed30 legacy=0x7f1a381d98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a2effd640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6793/0 conn(0x7f1a3811ed30 legacy=0x7f1a381d98f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.395 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 33046, 2024-09-13T13:51:49.396 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:49.397 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 27, 2024-09-13T13:51:49.398 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.399 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.400 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:49.401 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.402 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.403 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:49.404 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:49.405 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.406 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.407 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:49.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.409 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.410 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.411 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.412 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.413 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.414 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.415 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:49.418 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.419 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.420 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.422 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.423 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.424 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.425 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.426 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.427 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.428 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:49.429 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1a1c03fcd0 legacy=0x7f1a1c042190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 10 monclient: shutdown 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 >> v1:172.21.15.121:6792/0 conn(0x7f1a380a3600 legacy=0x7f1a381cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a3ea9b640 1 -- 172.21.15.121:0/3087631376 reap_dead start 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a3ea9b640 1 -- 172.21.15.121:0/3087631376 reap_dead start 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 shutdown_connections 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 >> 172.21.15.121:0/3087631376 conn(0x7f1a3806cee0 msgr2=0x7f1a3813fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 shutdown_connections 2024-09-13T13:51:49.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.388+0000 7f1a404fe640 1 -- 172.21.15.121:0/3087631376 wait complete. 2024-09-13T13:51:49.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:49.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-13T13:51:49.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-13T13:51:49.480 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: starting. 2024-09-13T13:51:49.480 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: creating pool ceph_test_rados_delete_pools_parallel.smithi153-34946 2024-09-13T13:51:49.480 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: created object 0... 2024-09-13T13:51:49.481 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: created object 25... 2024-09-13T13:51:49.481 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: created object 49... 2024-09-13T13:51:49.481 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: finishing. 2024-09-13T13:51:49.481 INFO:teuthology.orchestra.run.smithi153.stdout:process_1_[34947]: shutting down. 2024-09-13T13:51:49.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.531+0000 7efef3bd1640 10 monclient: get_monmap_and_config 2024-09-13T13:51:49.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.531+0000 7efef3bd1640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.531+0000 7efef3bd1640 10 monclient: monmap: 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:49.532754+0000 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:49.532754+0000 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.533 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.534 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.534 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.534 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.534 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.531+0000 7efef3bd1640 1 Processor -- start 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.531+0000 7efef3bd1640 1 -- start start 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.531+0000 7efef3bd1640 10 monclient: init 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient: _add_conns ranks=[2,3,6,7,5,0,8,1,4] 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient(hunting): picked mon.noname-b con 0x7efeec138bd0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient(hunting): picked mon.noname-g con 0x7efeec138fb0 addr v1:172.21.15.153:6790/0 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient(hunting): picked mon.noname-d con 0x7efeec139420 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec09e100 con 0x7efeec138bd0 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec09db40 con 0x7efeec138fb0 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec09de20 con 0x7efeec139420 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef3bd1640 10 monclient(hunting): authenticate will time out at 789.162299s 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef196d640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7efeec138bd0 0x7efeec0a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59258/0 (socket says 172.21.15.121:59258) 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef196d640 1 -- 172.21.15.121:0/3022842941 learned_addr learned my addr 172.21.15.121:0/3022842941 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.532+0000 7efef216e640 1 --1- 172.21.15.121:0/3022842941 >> v1:172.21.15.121:6792/0 conn(0x7efeec139420 0x7efeec141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52494/0 (socket says 172.21.15.121:52494) 2024-09-13T13:51:49.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef196d640 1 -- 172.21.15.121:0/3022842941 >> v1:172.21.15.121:6790/0 conn(0x7efeec138bd0 legacy=0x7efeec0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef216e640 1 -- 172.21.15.121:0/3022842941 >> v1:172.21.15.121:6792/0 conn(0x7efeec139420 legacy=0x7efeec141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2157481776 0 0) 0x7efeec09e100 con 0x7efeec138bd0 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): my global_id is 4430 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed4003520 con 0x7efeec138bd0 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355492907 0 0) 0x7efeec09de20 con 0x7efeec139420 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient(hunting): my global_id is 4470 2024-09-13T13:51:49.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec09e100 con 0x7efeec139420 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef116c640 1 -- 172.21.15.121:0/3022842941 >> v1:172.21.15.153:6790/0 conn(0x7efeec138fb0 legacy=0x7efeec13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2274234625 0 0) 0x7efed4003520 con 0x7efeec138bd0 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 >> v1:172.21.15.121:6792/0 conn(0x7efeec139420 legacy=0x7efeec141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 >> v1:172.21.15.153:6790/0 conn(0x7efeec138fb0 legacy=0x7efeec13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 monclient: found mon.noname-b 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec0a5da0 con 0x7efeec138bd0 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efee4001ae0 con 0x7efeec138bd0 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.534+0000 7efef3bd1640 5 monclient: authenticate success, global_id 4430 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: dump: 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.542 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efee40048a0 con 0x7efeec138bd0 2024-09-13T13:51:49.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 1 -- 172.21.15.121:0/3022842941 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efee4004e10 con 0x7efeec138bd0 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 4 set_mon_vals no callback set 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef096b640 10 monclient: dump: 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.544 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 10 monclient: shutdown 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 1 -- 172.21.15.121:0/3022842941 >> v1:172.21.15.121:6790/0 conn(0x7efeec138bd0 legacy=0x7efeec0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 1 -- 172.21.15.121:0/3022842941 shutdown_connections 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 1 -- 172.21.15.121:0/3022842941 >> 172.21.15.121:0/3022842941 conn(0x7efeec0ab800 msgr2=0x7efeec0abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 1 -- 172.21.15.121:0/3022842941 shutdown_connections 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.535+0000 7efef3bd1640 1 -- 172.21.15.121:0/3022842941 wait complete. 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.536+0000 7efef3bd1640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.536+0000 7efef3bd1640 10 monclient: monmap: 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:49.546 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.536+0000 7efef3bd1640 1 Processor -- start 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.536+0000 7efef3bd1640 1 -- start start 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.536+0000 7efef3bd1640 10 monclient: init 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient: _add_conns ranks=[0,5,4,8,7,1,3,2,6] 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient(hunting): picked mon.noname-a con 0x7efeec138bd0 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient(hunting): picked mon.noname-h con 0x7efeec138fb0 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient(hunting): picked mon.noname-c con 0x7efeec139420 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0a5b40 con 0x7efeec138bd0 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0a4cf0 con 0x7efeec139420 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec13b2f0 con 0x7efeec138fb0 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef3bd1640 10 monclient(hunting): authenticate will time out at 789.167181s 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef196d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efeec138bd0 0x7efeec0a32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45608/0 (socket says 172.21.15.121:45608) 2024-09-13T13:51:49.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.537+0000 7efef196d640 1 -- 172.21.15.121:0/4181166182 learned_addr learned my addr 172.21.15.121:0/4181166182 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.539+0000 7efef196d640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.121:6789/0 conn(0x7efeec138bd0 legacy=0x7efeec0a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.539+0000 7efef216e640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.121:6791/0 conn(0x7efeec139420 legacy=0x7efeec0a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.539+0000 7efef116c640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6791/0 conn(0x7efeec138fb0 legacy=0x7efeec0a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.539+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 DispatchQueue::entry inject delay of 0.002000 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 878104263 0 0) 0x7efeec0a5b40 con 0x7efeec138bd0 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): my global_id is 4500 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed8003520 con 0x7efeec138bd0 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179387760 0 0) 0x7efeec13b2f0 con 0x7efeec138fb0 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): my global_id is 4397 2024-09-13T13:51:49.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec0a5b40 con 0x7efeec138fb0 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047623099 0 0) 0x7efeec0a4cf0 con 0x7efeec139420 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): my global_id is 4396 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec13b2f0 con 0x7efeec139420 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3459474929 0 0) 0x7efed8003520 con 0x7efeec138bd0 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efeec0a4cf0 con 0x7efeec138bd0 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 614034033 0 0) 0x7efeec13b2f0 con 0x7efeec139420 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efed8003520 con 0x7efeec139420 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efee4004d90 con 0x7efeec138bd0 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.554 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efedc002090 con 0x7efeec139420 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.541+0000 7efeea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.556 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4202942706 0 0) 0x7efeec0a4cf0 con 0x7efeec138bd0 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6791/0 conn(0x7efeec138fb0 legacy=0x7efeec0a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.121:6791/0 conn(0x7efeec139420 legacy=0x7efeec0a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 monclient: found mon.a 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec13aa50 con 0x7efeec138bd0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 5 monclient: authenticate success, global_id 4500 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 10 monclient: _renew_subs 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efeec142940 con 0x7efeec138bd0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 10 monclient: _renew_subs 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efeec1d6fe0 con 0x7efeec138bd0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efee4001d30 con 0x7efeec138bd0 2024-09-13T13:51:49.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efee40048a0 con 0x7efeec138bd0 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeebfff640 4 set_mon_vals no callback set 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 10 monclient: dump: 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.559 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.542+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7efee4011c20 con 0x7efeec138bd0 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.543+0000 7efef3bd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.543+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efecc0065c0 con 0x7efecc002860 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.543+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 3898+0+0 (unknown 566398313 0 0) 0x7efee404e350 con 0x7efeec138bd0 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.543+0000 7efef216e640 10 monclient: get_auth_request con 0x7efed8040400 auth_method 0 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efef116c640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6789/0 conn(0x7efecc002860 legacy=0x7efecc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047883862 0 0) 0x7efecc0065c0 con 0x7efecc002860 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 10 monclient(hunting): my global_id is 4411 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec0a4cf0 con 0x7efecc002860 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3069869237 0 0) 0x7efeec0a4cf0 con 0x7efecc002860 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efecc0065c0 con 0x7efecc002860 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efee0001cb0 con 0x7efecc002860 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.545+0000 7efeea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 10 monclient: dump: 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.561 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1717204141 0 0) 0x7efecc0065c0 con 0x7efecc002860 2024-09-13T13:51:49.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efecc00a160 con 0x7efecc002860 2024-09-13T13:51:49.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7efee00068b0 con 0x7efecc002860 2024-09-13T13:51:49.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:49.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6789/0 conn(0x7efecc002860 legacy=0x7efecc004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.546+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6789/0 conn(0x7efecc002860 legacy=0x7efecc004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.552+0000 7efef3bd1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.552+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7efecc009e50 con 0x7efecc009110 2024-09-13T13:51:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efef116c640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6789/0 conn(0x7efecc009110 legacy=0x7efecc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3354408244 0 0) 0x7efecc009e50 con 0x7efecc009110 2024-09-13T13:51:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient(hunting): my global_id is 4420 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efecc0065c0 con 0x7efecc009110 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1242380885 0 0) 0x7efecc0065c0 con 0x7efecc009110 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efecc009e50 con 0x7efecc009110 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7efee0003d80 con 0x7efecc009110 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 10 monclient: dump: 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.565 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1735919000 0 0) 0x7efecc009e50 con 0x7efecc009110 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.555+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 --> v1:172.21.15.153:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efecc00c3f0 con 0x7efecc009110 2024-09-13T13:51:49.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 <== mon.1 v1:172.21.15.153:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 2314830820) 0x7efee0006b20 con 0x7efecc009110 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efeea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efeea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6789/0 conn(0x7efecc009110 legacy=0x7efecc0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efeea7fc640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6789/0 conn(0x7efecc009110 legacy=0x7efecc0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:49.567 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 33208, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.568 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 28, 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.569 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.570 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:49.571 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.572 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.574 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.575 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.578 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.579 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:49.580 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.581 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.582 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.583 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.584 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.585 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.586 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.587 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:49.587 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:49.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.153:6812/1206688214 conn(0x7efed8040400 legacy=0x7efed80428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 10 monclient: shutdown 2024-09-13T13:51:49.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 >> v1:172.21.15.121:6789/0 conn(0x7efeec138bd0 legacy=0x7efeec0a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef216e640 1 -- 172.21.15.121:0/4181166182 reap_dead start 2024-09-13T13:51:49.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef216e640 1 -- 172.21.15.121:0/4181166182 reap_dead start 2024-09-13T13:51:49.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 shutdown_connections 2024-09-13T13:51:49.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 >> 172.21.15.121:0/4181166182 conn(0x7efeec0ab800 msgr2=0x7efeec135ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 shutdown_connections 2024-09-13T13:51:49.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.556+0000 7efef3bd1640 1 -- 172.21.15.121:0/4181166182 wait complete. 2024-09-13T13:51:49.589 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:49.589 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-13T13:51:49.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.685+0000 7f2ac7aa2640 10 monclient: get_monmap_and_config 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.685+0000 7f2ac7aa2640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.685+0000 7f2ac7aa2640 10 monclient: monmap: 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:49.687421+0000 2024-09-13T13:51:49.687 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:49.687421+0000 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.688 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 1 Processor -- start 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 1 -- start start 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 10 monclient: init 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 10 monclient: _add_conns ranks=[0,7,5,6,8,2,1,4,3] 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac00a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.686+0000 7f2ac7aa2640 10 monclient(hunting): picked mon.noname-i con 0x7f2ac009d150 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac7aa2640 10 monclient(hunting): picked mon.noname-h con 0x7f2ac00a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac7aa2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac009e210 con 0x7f2ac00a6160 2024-09-13T13:51:49.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac7aa2640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac00a7ef0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac7aa2640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac009d970 con 0x7f2ac009d150 2024-09-13T13:51:49.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac7aa2640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac7aa2640 10 monclient(hunting): authenticate will time out at 789.317036s 2024-09-13T13:51:49.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac583e640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2ac00a6160 0x7f2ac00a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:45618/0 (socket says 172.21.15.121:45618) 2024-09-13T13:51:49.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.687+0000 7f2ac583e640 1 -- 172.21.15.121:0/1052305415 learned_addr learned my addr 172.21.15.121:0/1052305415 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac583e640 1 -- 172.21.15.121:0/1052305415 >> v1:172.21.15.121:6789/0 conn(0x7f2ac00a6160 legacy=0x7f2ac00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac603f640 1 -- 172.21.15.121:0/1052305415 >> v1:172.21.15.153:6791/0 conn(0x7f2ac00a6c50 legacy=0x7f2ac0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac503d640 1 -- 172.21.15.121:0/1052305415 >> v1:172.21.15.153:6792/0 conn(0x7f2ac009d150 legacy=0x7f2ac013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19660771 0 0) 0x7f2ac009e210 con 0x7f2ac00a6160 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): my global_id is 4509 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aac003520 con 0x7f2ac00a6160 2024-09-13T13:51:49.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742311110 0 0) 0x7f2ac00a7ef0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): my global_id is 4406 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac009e210 con 0x7f2ac00a6c50 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 697777407 0 0) 0x7f2ac009d970 con 0x7f2ac009d150 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient(hunting): my global_id is 4453 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac00a7ef0 con 0x7f2ac009d150 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 773401692 0 0) 0x7f2aac003520 con 0x7f2ac00a6160 2024-09-13T13:51:49.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 >> v1:172.21.15.153:6792/0 conn(0x7f2ac009d150 legacy=0x7f2ac013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 >> v1:172.21.15.153:6791/0 conn(0x7f2ac00a6c50 legacy=0x7f2ac0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 monclient: found mon.noname-a 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac009e630 con 0x7f2ac00a6160 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2abc001ae0 con 0x7f2ac00a6160 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac7aa2640 5 monclient: authenticate success, global_id 4509 2024-09-13T13:51:49.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac7aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: dump: 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.698 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac7aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2abc0048a0 con 0x7f2ac00a6160 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac7aa2640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 1 -- 172.21.15.121:0/1052305415 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2abc004e10 con 0x7f2ac00a6160 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac7aa2640 4 set_mon_vals no callback set 2024-09-13T13:51:49.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.689+0000 7f2ac483c640 10 monclient: dump: 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.700 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 10 monclient: shutdown 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/1052305415 >> v1:172.21.15.121:6789/0 conn(0x7f2ac00a6160 legacy=0x7f2ac00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/1052305415 shutdown_connections 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/1052305415 >> 172.21.15.121:0/1052305415 conn(0x7f2ac00ab940 msgr2=0x7f2ac00abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/1052305415 shutdown_connections 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/1052305415 wait complete. 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.690+0000 7f2ac7aa2640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient: monmap: 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.701 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.702 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 1 Processor -- start 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 1 -- start start 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient: init 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient: _add_conns ranks=[4,5,0,8,6,2,7,3,1] 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac009d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient(hunting): picked mon.noname-h con 0x7f2ac00a6160 addr v1:172.21.15.153:6791/0 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f2ac00a6c50 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac01ce3d0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac0126000 con 0x7f2ac009d150 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac0124c30 con 0x7f2ac00a6160 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac7aa2640 10 monclient(hunting): authenticate will time out at 789.321885s 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac583e640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f2ac009d150 0x7f2ac01cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33888/0 (socket says 172.21.15.121:33888) 2024-09-13T13:51:49.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.691+0000 7f2ac583e640 1 -- 172.21.15.121:0/2991547446 learned_addr learned my addr 172.21.15.121:0/2991547446 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac583e640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.121:6791/0 conn(0x7f2ac009d150 legacy=0x7f2ac01cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac603f640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.121:6789/0 conn(0x7f2ac00a6c50 legacy=0x7f2ac01d5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac503d640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6791/0 conn(0x7f2ac00a6160 legacy=0x7f2ac01394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492279517 0 0) 0x7f2ac01ce3d0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): my global_id is 4518 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa4003520 con 0x7f2ac00a6c50 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458508695 0 0) 0x7f2ac0126000 con 0x7f2ac009d150 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): my global_id is 4405 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac01ce3d0 con 0x7f2ac009d150 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890574324 0 0) 0x7f2ac0124c30 con 0x7f2ac00a6160 2024-09-13T13:51:49.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): my global_id is 4415 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac0126000 con 0x7f2ac00a6160 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 265864042 0 0) 0x7f2aa4003520 con 0x7f2ac00a6c50 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ac0124c30 con 0x7f2ac00a6c50 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2ab8002070 con 0x7f2ac00a6c50 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 604565394 0 0) 0x7f2ac01ce3d0 con 0x7f2ac009d150 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2aa4003520 con 0x7f2ac009d150 2024-09-13T13:51:49.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2abc004d90 con 0x7f2ac009d150 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.712 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 698779522 0 0) 0x7f2ac0126000 con 0x7f2ac00a6160 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2ac01ce3d0 con 0x7f2ac00a6160 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2ab0003610 con 0x7f2ac00a6160 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.713 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1193372707 0 0) 0x7f2ac0124c30 con 0x7f2ac00a6c50 2024-09-13T13:51:49.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6791/0 conn(0x7f2ac00a6160 legacy=0x7f2ac01394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.121:6791/0 conn(0x7f2ac009d150 legacy=0x7f2ac01cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 monclient: found mon.a 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac0126f90 con 0x7f2ac00a6c50 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 5 monclient: authenticate success, global_id 4518 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 10 monclient: _renew_subs 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac01266f0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 10 monclient: _renew_subs 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2ac0139dc0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2ab8003c40 con 0x7f2ac00a6c50 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2ab80041b0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab7fff640 4 set_mon_vals no callback set 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.694+0000 7f2ab67fc640 10 monclient: dump: 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.695+0000 7f2ac7aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.695+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2aa80065c0 con 0x7f2aa8002860 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.695+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f2ab8004350 con 0x7f2ac00a6c50 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.695+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(23..23 src has 1..23) ==== 3898+0+0 (unknown 566398313 0 0) 0x7f2ab804e4f0 con 0x7f2ac00a6c50 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.696+0000 7f2ac503d640 10 monclient: get_auth_request con 0x7f2aa4040530 auth_method 0 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.697+0000 7f2ac583e640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6790/0 conn(0x7f2aa8002860 legacy=0x7f2aa8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639007378 0 0) 0x7f2aa80065c0 con 0x7f2aa8002860 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient(hunting): my global_id is 4422 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac0124c30 con 0x7f2aa8002860 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 309550191 0 0) 0x7f2ac0124c30 con 0x7f2aa8002860 2024-09-13T13:51:49.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2aa80065c0 con 0x7f2aa8002860 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2abc001d30 con 0x7f2aa8002860 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient: dump: 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2211271870 0 0) 0x7f2aa80065c0 con 0x7f2aa8002860 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2aa800a160 con 0x7f2aa8002860 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2abc005840 con 0x7f2aa8002860 2024-09-13T13:51:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6790/0 conn(0x7f2aa8002860 legacy=0x7f2aa8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.698+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6790/0 conn(0x7f2aa8002860 legacy=0x7f2aa8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.704+0000 7f2ac7aa2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.704+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2aa8009e70 con 0x7f2aa8009130 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ac583e640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6790/0 conn(0x7f2aa8009130 legacy=0x7f2aa80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282611134 0 0) 0x7f2aa8009e70 con 0x7f2aa8009130 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient(hunting): my global_id is 4431 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa80065c0 con 0x7f2aa8009130 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1359506851 0 0) 0x7f2aa80065c0 con 0x7f2aa8009130 2024-09-13T13:51:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2aa8009e70 con 0x7f2aa8009130 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f2abc0060b0 con 0x7f2aa8009130 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 10 monclient: dump: 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2772349747 0 0) 0x7f2aa8009e70 con 0x7f2aa8009130 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.707+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 --> v1:172.21.15.153:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2aa800c410 con 0x7f2aa8009130 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 <== mon.3 v1:172.21.15.153:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7559 (unknown 0 0 1535058882) 0x7f2abc008330 con 0x7f2aa8009130 2024-09-13T13:51:49.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ab67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ab67fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6790/0 conn(0x7f2aa8009130 legacy=0x7f2aa80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ab67fc640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6790/0 conn(0x7f2aa8009130 legacy=0x7f2aa80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 33332, 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:49.725 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 28, 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.726 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.727 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:49.728 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.729 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.730 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.732 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.733 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.734 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.736 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.737 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.738 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.739 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.740 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.741 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.742 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.743 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.744 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:49.744 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.153:6812/1206688214 conn(0x7f2aa4040530 legacy=0x7f2aa40429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 10 monclient: shutdown 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 >> v1:172.21.15.121:6789/0 conn(0x7f2ac00a6c50 legacy=0x7f2ac01d5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac603f640 1 -- 172.21.15.121:0/2991547446 reap_dead start 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac603f640 1 -- 172.21.15.121:0/2991547446 reap_dead start 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 shutdown_connections 2024-09-13T13:51:49.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 >> 172.21.15.121:0/2991547446 conn(0x7f2ac00ab940 msgr2=0x7f2ac009ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 shutdown_connections 2024-09-13T13:51:49.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.708+0000 7f2ac7aa2640 1 -- 172.21.15.121:0/2991547446 wait complete. 2024-09-13T13:51:49.746 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:49.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-13T13:51:49.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-13T13:51:49.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.842+0000 7fc04c0bf640 10 monclient: get_monmap_and_config 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.842+0000 7fc04c0bf640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.842+0000 7fc04c0bf640 10 monclient: monmap: 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:49.843897+0000 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:49.843897+0000 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.844 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.845 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.842+0000 7fc04c0bf640 1 Processor -- start 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.842+0000 7fc04c0bf640 1 -- start start 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.842+0000 7fc04c0bf640 10 monclient: init 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient: _add_conns ranks=[6,2,4,7,3,0,5,1,8] 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient(hunting): picked mon.noname-d con 0x7fc044138bd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient(hunting): picked mon.noname-b con 0x7fc044138fb0 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient(hunting): picked mon.noname-c con 0x7fc044139420 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04409e100 con 0x7fc044138fb0 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04409db40 con 0x7fc044139420 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04409de20 con 0x7fc044138bd0 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04c0bf640 10 monclient(hunting): authenticate will time out at 789.473461s 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04965a640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fc044138fb0 0x7fc04413df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59268/0 (socket says 172.21.15.121:59268) 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc049e5b640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fc044138bd0 0x7fc0440a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52506/0 (socket says 172.21.15.121:52506) 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04965a640 1 -- 172.21.15.121:0/655701259 learned_addr learned my addr 172.21.15.121:0/655701259 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.843+0000 7fc04a65c640 1 --1- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6791/0 conn(0x7fc044139420 0x7fc044141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33900/0 (socket says 172.21.15.121:33900) 2024-09-13T13:51:49.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc04a65c640 1 -- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6791/0 conn(0x7fc044139420 legacy=0x7fc044141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc049e5b640 1 -- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6792/0 conn(0x7fc044138bd0 legacy=0x7fc0440a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc04965a640 1 -- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6790/0 conn(0x7fc044138fb0 legacy=0x7fc04413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1608280762 0 0) 0x7fc04409e100 con 0x7fc044138fb0 2024-09-13T13:51:49.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): my global_id is 4439 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc02c003520 con 0x7fc044138fb0 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423300407 0 0) 0x7fc04409de20 con 0x7fc044138bd0 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): my global_id is 4479 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04409e100 con 0x7fc044138bd0 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3729392530 0 0) 0x7fc04409db40 con 0x7fc044139420 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 10 monclient(hunting): my global_id is 4414 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.845+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc04409de20 con 0x7fc044139420 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3051586084 0 0) 0x7fc04409e100 con 0x7fc044138bd0 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6791/0 conn(0x7fc044139420 legacy=0x7fc044141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6790/0 conn(0x7fc044138fb0 legacy=0x7fc04413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 monclient: found mon.noname-d 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0440a5da0 con 0x7fc044138bd0 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc03c0019f0 con 0x7fc044138bd0 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: dump: 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.854 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 5 monclient: authenticate success, global_id 4479 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc03c001cc0 con 0x7fc044138bd0 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 1 -- 172.21.15.121:0/655701259 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc03c004b20 con 0x7fc044138bd0 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 4 set_mon_vals no callback set 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc048e59640 10 monclient: dump: 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.856 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 10 monclient: shutdown 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 1 -- 172.21.15.121:0/655701259 >> v1:172.21.15.121:6792/0 conn(0x7fc044138bd0 legacy=0x7fc0440a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 1 -- 172.21.15.121:0/655701259 shutdown_connections 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 1 -- 172.21.15.121:0/655701259 >> 172.21.15.121:0/655701259 conn(0x7fc0440ab800 msgr2=0x7fc0440abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.846+0000 7fc04c0bf640 1 -- 172.21.15.121:0/655701259 shutdown_connections 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.847+0000 7fc04c0bf640 1 -- 172.21.15.121:0/655701259 wait complete. 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.847+0000 7fc04c0bf640 10 monclient: build_initial_monmap 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.847+0000 7fc04c0bf640 10 monclient: monmap: 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:49.857 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.847+0000 7fc04c0bf640 1 Processor -- start 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.847+0000 7fc04c0bf640 1 -- start start 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.847+0000 7fc04c0bf640 10 monclient: init 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient: _add_conns ranks=[7,6,2,8,1,4,5,0,3] 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient(hunting): picked mon.noname-i con 0x7fc044138bd0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient(hunting): picked mon.noname-d con 0x7fc044138fb0 addr v1:172.21.15.121:6792/0 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient(hunting): picked mon.noname-b con 0x7fc044139420 addr v1:172.21.15.121:6790/0 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0440a5590 con 0x7fc044139420 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0440a58b0 con 0x7fc044138fb0 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc04412f630 con 0x7fc044138bd0 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04c0bf640 10 monclient(hunting): authenticate will time out at 789.478373s 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04965a640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fc044138fb0 0x7fc04412ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:52514/0 (socket says 172.21.15.121:52514) 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04965a640 1 -- 172.21.15.121:0/579683218 learned_addr learned my addr 172.21.15.121:0/579683218 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:49.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.848+0000 7fc04a65c640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fc044139420 0x7fc0441d5fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:59278/0 (socket says 172.21.15.121:59278) 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc04a65c640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.121:6790/0 conn(0x7fc044139420 legacy=0x7fc0441d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc04965a640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.121:6792/0 conn(0x7fc044138fb0 legacy=0x7fc04412ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1751878858 0 0) 0x7fc0440a5590 con 0x7fc044139420 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): my global_id is 4448 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc030003520 con 0x7fc044139420 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314124489 0 0) 0x7fc0440a58b0 con 0x7fc044138fb0 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): my global_id is 4488 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0440a5590 con 0x7fc044138fb0 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc049e5b640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6792/0 conn(0x7fc044138bd0 legacy=0x7fc04412e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 32266235 0 0) 0x7fc030003520 con 0x7fc044139420 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0440a58b0 con 0x7fc044139420 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc034001c50 con 0x7fc044139420 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.865 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3523170059 0 0) 0x7fc04412f630 con 0x7fc044138bd0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 10 monclient(hunting): my global_id is 4462 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.850+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc030003520 con 0x7fc044138bd0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2075896725 0 0) 0x7fc0440a5590 con 0x7fc044138fb0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc04412f630 con 0x7fc044138fb0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc038004b60 con 0x7fc044138fb0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient(hunting): dump: 2024-09-13T13:51:49.867 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.868 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1658444261 0 0) 0x7fc0440a58b0 con 0x7fc044139420 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6792/0 conn(0x7fc044138bd0 legacy=0x7fc04412e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.121:6792/0 conn(0x7fc044138fb0 legacy=0x7fc04412ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 monclient: found mon.b 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc044131f60 con 0x7fc044139420 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:49.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 5 monclient: authenticate success, global_id 4448 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 10 monclient: _renew_subs 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc044130270 con 0x7fc044139420 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 10 monclient: _renew_subs 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc044130f00 con 0x7fc044139420 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc034004a90 con 0x7fc044139420 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc034005000 con 0x7fc044139420 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc043fff640 4 set_mon_vals no callback set 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.851+0000 7fc0427fc640 10 monclient: dump: 2024-09-13T13:51:49.870 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.871 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.852+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fc0340051a0 con 0x7fc044139420 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.852+0000 7fc04c0bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.852+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc0240065c0 con 0x7fc024002860 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.852+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(23..23 src has 1..23) ==== 3898+0+0 (unknown 566398313 0 0) 0x7fc03404e1b0 con 0x7fc044139420 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.852+0000 7fc04965a640 10 monclient: get_auth_request con 0x7fc0300403c0 auth_method 0 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc049e5b640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6791/0 conn(0x7fc024002860 legacy=0x7fc024004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2489244102 0 0) 0x7fc0240065c0 con 0x7fc024002860 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient(hunting): my global_id is 4424 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0440a58b0 con 0x7fc024002860 2024-09-13T13:51:49.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 515320318 0 0) 0x7fc0440a58b0 con 0x7fc024002860 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc0240065c0 con 0x7fc024002860 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc03c004be0 con 0x7fc024002860 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 10 monclient: dump: 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.873 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2594631330 0 0) 0x7fc0240065c0 con 0x7fc024002860 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.854+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc02400a160 con 0x7fc024002860 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.855+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc03c0015a0 con 0x7fc024002860 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.855+0000 7fc0427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.855+0000 7fc0427fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.855+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6791/0 conn(0x7fc024002860 legacy=0x7fc024004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.855+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6791/0 conn(0x7fc024002860 legacy=0x7fc024004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.861+0000 7fc04c0bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:49.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.861+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc024009e70 con 0x7fc024009130 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc049e5b640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6791/0 conn(0x7fc024009130 legacy=0x7fc0240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193287941 0 0) 0x7fc024009e70 con 0x7fc024009130 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc0427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc0427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc0427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc0427fc640 10 monclient(hunting): my global_id is 4433 2024-09-13T13:51:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.863+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0240065c0 con 0x7fc024009130 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2356619886 0 0) 0x7fc0240065c0 con 0x7fc024009130 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc024009e70 con 0x7fc024009130 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fc03c004e60 con 0x7fc024009130 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 10 monclient: dump: 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:49.877 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 697145644 0 0) 0x7fc024009e70 con 0x7fc024009130 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 --> v1:172.21.15.153:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc02400c410 con 0x7fc024009130 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 <== mon.5 v1:172.21.15.153:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7893 (unknown 0 0 3517698702) 0x7fc03c007420 con 0x7fc024009130 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6791/0 conn(0x7fc024009130 legacy=0x7fc0240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.864+0000 7fc0427fc640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6791/0 conn(0x7fc024009130 legacy=0x7fc0240095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 33497, 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:49.879 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 28, 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:49.880 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.881 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:49.882 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.884 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.885 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.886 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.888 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.889 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.890 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:49.891 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.892 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.894 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:49.895 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.897 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.898 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.899 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:49.900 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.153:6812/1206688214 conn(0x7fc0300403c0 legacy=0x7fc030042880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 10 monclient: shutdown 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 >> v1:172.21.15.121:6790/0 conn(0x7fc044139420 legacy=0x7fc0441d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04a65c640 1 -- 172.21.15.121:0/579683218 reap_dead start 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04a65c640 1 -- 172.21.15.121:0/579683218 reap_dead start 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 shutdown_connections 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 >> 172.21.15.121:0/579683218 conn(0x7fc0440ab800 msgr2=0x7fc04412c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 shutdown_connections 2024-09-13T13:51:49.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.865+0000 7fc04c0bf640 1 -- 172.21.15.121:0/579683218 wait complete. 2024-09-13T13:51:49.903 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:49.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-13T13:51:49.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-13T13:51:50.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.999+0000 7fbde1524640 10 monclient: get_monmap_and_config 2024-09-13T13:51:50.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.999+0000 7fbde1524640 10 monclient: build_initial_monmap 2024-09-13T13:51:50.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:49.999+0000 7fbde1524640 10 monclient: monmap: 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:51:50.001496+0000 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:51:50.001496+0000 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:50.002 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:50.003 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:50.003 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.000+0000 7fbde1524640 1 Processor -- start 2024-09-13T13:51:50.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.000+0000 7fbde1524640 1 -- start start 2024-09-13T13:51:50.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.000+0000 7fbde1524640 10 monclient: init 2024-09-13T13:51:50.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.000+0000 7fbde1524640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.000+0000 7fbde1524640 10 monclient: _add_conns ranks=[4,7,8,0,5,3,1,2,6] 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.000+0000 7fbde1524640 10 monclient(hunting): picked mon.noname-c con 0x7fbddc138bd0 addr v1:172.21.15.121:6791/0 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 10 monclient(hunting): picked mon.noname-i con 0x7fbddc138fb0 addr v1:172.21.15.153:6792/0 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 10 monclient(hunting): picked mon.noname-e con 0x7fbddc139420 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc09e100 con 0x7fbddc138bd0 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc09db40 con 0x7fbddc138fb0 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc09de20 con 0x7fbddc139420 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbde1524640 10 monclient(hunting): authenticate will time out at 789.631076s 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbddaffd640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fbddc138bd0 0x7fbddc0a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33912/0 (socket says 172.21.15.121:33912) 2024-09-13T13:51:50.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.001+0000 7fbddaffd640 1 -- 172.21.15.121:0/1248610751 learned_addr learned my addr 172.21.15.121:0/1248610751 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:50.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbddaffd640 1 -- 172.21.15.121:0/1248610751 >> v1:172.21.15.121:6791/0 conn(0x7fbddc138bd0 legacy=0x7fbddc0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbddb7fe640 1 -- 172.21.15.121:0/1248610751 >> v1:172.21.15.121:6793/0 conn(0x7fbddc139420 legacy=0x7fbddc141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdd3fff640 1 -- 172.21.15.121:0/1248610751 >> v1:172.21.15.153:6792/0 conn(0x7fbddc138fb0 legacy=0x7fbddc13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856358389 0 0) 0x7fbddc09e100 con 0x7fbddc138bd0 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): my global_id is 4423 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc8003520 con 0x7fbddc138bd0 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881332041 0 0) 0x7fbddc09de20 con 0x7fbddc139420 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): my global_id is 4472 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbddc09e100 con 0x7fbddc139420 2024-09-13T13:51:50.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923335565 0 0) 0x7fbddc09db40 con 0x7fbddc138fb0 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient(hunting): my global_id is 4471 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbddc09de20 con 0x7fbddc138fb0 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2027664529 0 0) 0x7fbddc09e100 con 0x7fbddc139420 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 >> v1:172.21.15.153:6792/0 conn(0x7fbddc138fb0 legacy=0x7fbddc13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 >> v1:172.21.15.121:6791/0 conn(0x7fbddc138bd0 legacy=0x7fbddc0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 monclient: found mon.noname-e 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:51:50.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbddc0a5da0 con 0x7fbddc139420 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: _finish_auth 0 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdd4001dc0 con 0x7fbddc139420 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbde1524640 5 monclient: authenticate success, global_id 4472 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbde1524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: dump: 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.015 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbde1524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdd4002090 con 0x7fbddc139420 2024-09-13T13:51:50.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 1 -- 172.21.15.121:0/1248610751 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdd4003cd0 con 0x7fbddc139420 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbde1524640 10 monclient: get_monmap_and_config success 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbde1524640 4 set_mon_vals no callback set 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.003+0000 7fbdda7fc640 10 monclient: dump: 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.017 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 10 monclient: shutdown 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 1 -- 172.21.15.121:0/1248610751 >> v1:172.21.15.121:6793/0 conn(0x7fbddc139420 legacy=0x7fbddc141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 1 -- 172.21.15.121:0/1248610751 shutdown_connections 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 1 -- 172.21.15.121:0/1248610751 >> 172.21.15.121:0/1248610751 conn(0x7fbddc0ab800 msgr2=0x7fbddc0abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:50.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 1 -- 172.21.15.121:0/1248610751 shutdown_connections 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 1 -- 172.21.15.121:0/1248610751 wait complete. 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.004+0000 7fbde1524640 10 monclient: build_initial_monmap 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient: monmap: 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:51:50.019 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 1 Processor -- start 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 1 -- start start 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient: init 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient: _reopen_session rank -1 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient: _add_conns ranks=[8,1,0,6,7,5,2,3,4] 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient(hunting): picked mon.noname-e con 0x7fbddc138bd0 addr v1:172.21.15.121:6793/0 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient(hunting): picked mon.noname-f con 0x7fbddc138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:51:50.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient(hunting): picked mon.noname-a con 0x7fbddc139420 addr v1:172.21.15.121:6789/0 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc131bc0 con 0x7fbddc139420 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc0a75c0 con 0x7fbddc138fb0 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbddc0a59d0 con 0x7fbddc138bd0 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient(hunting): _renew_subs 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbde1524640 10 monclient(hunting): authenticate will time out at 789.635832s 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbddaffd640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fbddc138bd0 0x7fbddc12dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56732/0 (socket says 172.21.15.121:56732) 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.005+0000 7fbddaffd640 1 -- 172.21.15.121:0/59078989 learned_addr learned my addr 172.21.15.121:0/59078989 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.007+0000 7fbddaffd640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.121:6793/0 conn(0x7fbddc138bd0 legacy=0x7fbddc12dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbddb7fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.121:6789/0 conn(0x7fbddc139420 legacy=0x7fbddc1d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd3fff640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6789/0 conn(0x7fbddc138fb0 legacy=0x7fbddc12e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1804063935 0 0) 0x7fbdcc0015e0 con 0x7fbddc138bd0 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4481 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdbc003520 con 0x7fbddc138bd0 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2964239520 0 0) 0x7fbddc0a5da0 con 0x7fbddc139420 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4527 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdcc0015e0 con 0x7fbddc139420 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740526961 0 0) 0x7fbddc09de20 con 0x7fbddc138fb0 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4429 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbddc0a5da0 con 0x7fbddc138fb0 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 908661445 0 0) 0x7fbdbc003520 con 0x7fbddc138bd0 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbddc09de20 con 0x7fbddc138bd0 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdcc004b20 con 0x7fbddc138bd0 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): dump: 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.023 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3975712335 0 0) 0x7fbdcc0015e0 con 0x7fbddc139420 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdbc003520 con 0x7fbddc139420 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdd4003cd0 con 0x7fbddc139420 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient(hunting): dump: 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.025 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3426224766 0 0) 0x7fbddc0a5da0 con 0x7fbddc138fb0 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdcc0015e0 con 0x7fbddc138fb0 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 454104369 0 0) 0x7fbddc09de20 con 0x7fbddc138bd0 2024-09-13T13:51:50.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6789/0 conn(0x7fbddc138fb0 legacy=0x7fbddc12e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.121:6789/0 conn(0x7fbddc139420 legacy=0x7fbddc1d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: _finish_hunting 0 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 monclient: found mon.e 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbddc12f440 con 0x7fbddc138bd0 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: _finish_auth 0 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: _check_auth_tickets 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 5 monclient: authenticate success, global_id 4481 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 10 monclient: _renew_subs 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:50.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbddc12ea40 con 0x7fbddc138bd0 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 10 monclient: _renew_subs 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fbddc130210 con 0x7fbddc138bd0 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbdcc004f70 con 0x7fbddc138bd0 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdcc005460 con 0x7fbddc138bd0 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd9ffb640 4 set_mon_vals no callback set 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.008+0000 7fbdd37fe640 10 monclient: dump: 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.028 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.009+0000 7fbde1524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:50.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.009+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fbdcc011ae0 con 0x7fbddc138bd0 2024-09-13T13:51:50.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.009+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdc00065c0 con 0x7fbdc0002860 2024-09-13T13:51:50.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.009+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(23..23 src has 1..23) ==== 3898+0+0 (unknown 566398313 0 0) 0x7fbdcc04e5a0 con 0x7fbddc138bd0 2024-09-13T13:51:50.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.009+0000 7fbddb7fe640 10 monclient: get_auth_request con 0x7fbdbc0403e0 auth_method 0 2024-09-13T13:51:50.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.011+0000 7fbdd3fff640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6792/0 conn(0x7fbdc0002860 legacy=0x7fbdc0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.011+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015420211 0 0) 0x7fbdc00065c0 con 0x7fbdc0002860 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.011+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.011+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.011+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.011+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4480 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbddc09de20 con 0x7fbdc0002860 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2693548820 0 0) 0x7fbddc09de20 con 0x7fbdc0002860 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc00065c0 con 0x7fbdc0002860 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdc4003e50 con 0x7fbdc0002860 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 10 monclient: dump: 2024-09-13T13:51:50.033 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.034 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3985705129 0 0) 0x7fbdc00065c0 con 0x7fbdc0002860 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdc000a160 con 0x7fbdc0002860 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbdc4004070 con 0x7fbdc0002860 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6792/0 conn(0x7fbdc0002860 legacy=0x7fbdc0004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.012+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6792/0 conn(0x7fbdc0002860 legacy=0x7fbdc0004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.018+0000 7fbde1524640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.018+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbdc0009e70 con 0x7fbdc0009130 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd3fff640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6792/0 conn(0x7fbdc0009130 legacy=0x7fbdc00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204433306 0 0) 0x7fbdc0009e70 con 0x7fbdc0009130 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:51:50.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient(hunting): my global_id is 4489 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdc00065c0 con 0x7fbdc0009130 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 974724403 0 0) 0x7fbdc00065c0 con 0x7fbdc0009130 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbdc0009e70 con 0x7fbdc0009130 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbdc4005b00 con 0x7fbdc0009130 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient: dump: 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:51:50.036 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 759761302 0 0) 0x7fbdc0009e70 con 0x7fbdc0009130 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 --> v1:172.21.15.153:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbdc000c3f0 con 0x7fbdc0009130 2024-09-13T13:51:50.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 <== mon.7 v1:172.21.15.153:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 564102329) 0x7fbdc4007e40 con 0x7fbdc0009130 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.021+0000 7fbdd37fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6792/0 conn(0x7fbdc0009130 legacy=0x7fbdc00095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbdd37fe640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6792/0 conn(0x7fbdc0009130 legacy=0x7fbdc00095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 14, 2024-09-13T13:51:50.038 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 33661, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 28, 2024-09-13T13:51:50.039 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:50.040 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:50.041 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:51:50.042 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:51:50.043 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.045 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.046 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.049 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.050 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.052 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:51:50.053 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.054 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:50.056 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:51:50.057 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:51:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:51:50.058 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:51:50.058 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.153:6812/1206688214 conn(0x7fbdbc0403e0 legacy=0x7fbdbc0428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 10 monclient: shutdown 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 >> v1:172.21.15.121:6793/0 conn(0x7fbddc138bd0 legacy=0x7fbddc12dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbddb7fe640 1 -- 172.21.15.121:0/59078989 reap_dead start 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbddb7fe640 1 -- 172.21.15.121:0/59078989 reap_dead start 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 shutdown_connections 2024-09-13T13:51:50.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 >> 172.21.15.121:0/59078989 conn(0x7fbddc0ab800 msgr2=0x7fbddc1373f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:51:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 shutdown_connections 2024-09-13T13:51:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:51:50.022+0000 7fbde1524640 1 -- 172.21.15.121:0/59078989 wait complete. 2024-09-13T13:51:50.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:51:50.060 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['f', 'i'] 2024-09-13T13:51:50.060 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['g'] 2024-09-13T13:51:50.060 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-09-13T13:51:50.061 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-09-13T13:51:50.061 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-09-13T13:51:50.061 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:51:50.071 INFO:tasks.ceph.mon.f:Stopped 2024-09-13T13:51:50.071 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-09-13T13:51:50.071 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-09-13T13:51:50.072 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-13T13:51:50.072 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:51:50.123 INFO:tasks.ceph.mon.i:Stopped 2024-09-13T13:51:50.123 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-09-13T13:51:50.123 INFO:tasks.ceph.mon.g:Sent signal 19 2024-09-13T13:51:50.123 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-13T13:52:04.397 INFO:teuthology.orchestra.run.smithi153.stdout:process_2_[34948]: starting. 2024-09-13T13:52:04.397 INFO:teuthology.orchestra.run.smithi153.stdout:process_2_[34948]: deleting pool ceph_test_rados_delete_pools_parallel.smithi153-34946 2024-09-13T13:52:04.397 INFO:teuthology.orchestra.run.smithi153.stdout:process_2_[34948]: shutting down. 2024-09-13T13:52:07.530 INFO:teuthology.orchestra.run.smithi153.stdout:******************************* 2024-09-13T13:52:07.530 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: starting. 2024-09-13T13:52:07.530 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: creating pool ceph_test_rados_delete_pools_parallel.smithi153-34946 2024-09-13T13:52:07.530 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: created object 0... 2024-09-13T13:52:07.531 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: created object 25... 2024-09-13T13:52:07.531 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: created object 49... 2024-09-13T13:52:07.531 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: finishing. 2024-09-13T13:52:07.531 INFO:teuthology.orchestra.run.smithi153.stdout:process_3_[34981]: shutting down. 2024-09-13T13:52:07.573 INFO:teuthology.orchestra.run.smithi153.stdout:******************************* 2024-09-13T13:52:07.573 INFO:teuthology.orchestra.run.smithi153.stdout:process_5_[34983]: starting. 2024-09-13T13:52:07.573 INFO:teuthology.orchestra.run.smithi153.stdout:process_5_[34983]: listing objects. 2024-09-13T13:52:07.573 INFO:teuthology.orchestra.run.smithi153.stdout:process_5_[34983]: listed object 0... 2024-09-13T13:52:07.573 INFO:teuthology.orchestra.run.smithi153.stdout:process_5_[34983]: listed object 25... 2024-09-13T13:52:07.573 INFO:teuthology.orchestra.run.smithi153.stdout:process_5_[34983]: saw 50 objects 2024-09-13T13:52:07.574 INFO:teuthology.orchestra.run.smithi153.stdout:process_5_[34983]: shutting down. 2024-09-13T13:52:08.408 INFO:teuthology.orchestra.run.smithi153.stdout:******************************* 2024-09-13T13:52:08.408 INFO:teuthology.orchestra.run.smithi153.stdout:process_4_[34982]: starting. 2024-09-13T13:52:08.408 INFO:teuthology.orchestra.run.smithi153.stdout:process_4_[34982]: deleting pool ceph_test_rados_delete_pools_parallel.smithi153-34946 2024-09-13T13:52:08.408 INFO:teuthology.orchestra.run.smithi153.stdout:process_4_[34982]: shutting down. 2024-09-13T13:52:08.410 INFO:teuthology.orchestra.run.smithi153.stdout:******************************* 2024-09-13T13:52:08.410 INFO:teuthology.orchestra.run.smithi153.stdout:******************************* 2024-09-13T13:52:08.410 INFO:teuthology.orchestra.run.smithi153.stdout:******* SUCCESS ********** 2024-09-13T13:52:08.412 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-13T13:52:08.422 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-13T13:52:10.124 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-09-13T13:52:10.125 INFO:tasks.ceph.mon.g:Sent signal 18 2024-09-13T13:52:10.125 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-09-13T13:52:10.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.231+0000 7fa90a9d5640 10 monclient: get_monmap_and_config 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.231+0000 7fa90a9d5640 10 monclient: build_initial_monmap 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.231+0000 7fa90a9d5640 10 monclient: monmap: 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:10.231916+0000 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:10.231916+0000 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:10.232 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:10.233 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.231+0000 7fa90a9d5640 1 Processor -- start 2024-09-13T13:52:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 1 -- start start 2024-09-13T13:52:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient: init 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient: _add_conns ranks=[3,8,2,7,4,5,0,1,6] 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient(hunting): picked mon.noname-g con 0x7fa904138c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient(hunting): picked mon.noname-e con 0x7fa90413bfc0 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient(hunting): picked mon.noname-b con 0x7fa90413fad0 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9040a5e80 con 0x7fa90413fad0 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa904139a60 con 0x7fa904138c50 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90409df00 con 0x7fa90413bfc0 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa90a9d5640 10 monclient(hunting): authenticate will time out at 809.861583s 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa908f72640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fa90413fad0 0x7fa904141ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:47940/0 (socket says 172.21.15.121:47940) 2024-09-13T13:52:10.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.232+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 learned_addr learned my addr 172.21.15.121:0/3149609600 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.234+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.121:6790/0 conn(0x7fa90413fad0 legacy=0x7fa904141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.234+0000 7fa9037fe640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.121:6793/0 conn(0x7fa90413bfc0 legacy=0x7fa90413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:10.235+0000 7fa903fff640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6790/0 conn(0x7fa904138c50 legacy=0x7fa9040a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): tick 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 monclient(hunting): continuing hunt 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.121:6793/0 conn(0x7fa90413bfc0 legacy=0x7fa90413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6790/0 conn(0x7fa904138c50 legacy=0x7fa9040a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.121:6790/0 conn(0x7fa90413fad0 legacy=0x7fa904141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient: _add_conns ranks=[1,4,7,5,6,0] 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): picked mon.noname-f con 0x7fa8e4003990 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e4007620 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:13.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): picked mon.noname-i con 0x7fa8e400b3b0 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 -- 172.21.15.121:0/3149609600 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90409d940 con 0x7fa8e4003990 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 -- 172.21.15.121:0/3149609600 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e4010150 con 0x7fa8e4007620 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 1 -- 172.21.15.121:0/3149609600 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e40113f0 con 0x7fa8e400b3b0 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa901ffb640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6792/0 conn(0x7fa8e400b3b0 legacy=0x7fa8e400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.232+0000 7fa903fff640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6789/0 conn(0x7fa8e4003990 legacy=0x7fa8e4005de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:13.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.235+0000 7fa9037fe640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.121:6791/0 conn(0x7fa8e4007620 legacy=0x7fa8e4009ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:13.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.433+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6792/0 conn(0x7fa8e400b3b0 legacy=0x7fa8e400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:13.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.433+0000 7fa903fff640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6789/0 conn(0x7fa8e4003990 legacy=0x7fa8e4005de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.835+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6792/0 conn(0x7fa8e400b3b0 legacy=0x7fa8e400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:13.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:13.835+0000 7fa903fff640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6789/0 conn(0x7fa8e4003990 legacy=0x7fa8e4005de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:14.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:14.637+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6792/0 conn(0x7fa8e400b3b0 legacy=0x7fa8e400d830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:14.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:14.637+0000 7fa903fff640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6789/0 conn(0x7fa8e4003990 legacy=0x7fa8e4005de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594714054 0 0) 0x7fa8e4010150 con 0x7fa8e4007620 2024-09-13T13:52:15.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient(hunting): my global_id is 4450 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f0003520 con 0x7fa8e4007620 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 902198810 0 0) 0x7fa8f0003520 con 0x7fa8e4007620 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6792/0 conn(0x7fa8e400b3b0 legacy=0x7fa8e400d830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.153:6789/0 conn(0x7fa8e4003990 legacy=0x7fa8e4005de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 monclient: found mon.noname-c 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa908f72640 1 -- 172.21.15.121:0/3149609600 reap_dead start 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:52:15.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e40126f0 con 0x7fa8e4007620 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8ec0019b0 con 0x7fa8e4007620 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa90a9d5640 5 monclient: authenticate success, global_id 4450 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa90a9d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: dump: 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.152 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa90a9d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8ec001c50 con 0x7fa8e4007620 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 1 -- 172.21.15.121:0/3149609600 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8ec003950 con 0x7fa8e4007620 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa902ffd640 10 monclient: dump: 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.154 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa90a9d5640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.148+0000 7fa90a9d5640 4 set_mon_vals no callback set 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 10 monclient: shutdown 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 1 -- 172.21.15.121:0/3149609600 >> v1:172.21.15.121:6791/0 conn(0x7fa8e4007620 legacy=0x7fa8e4009ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 1 -- 172.21.15.121:0/3149609600 shutdown_connections 2024-09-13T13:52:15.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 1 -- 172.21.15.121:0/3149609600 >> 172.21.15.121:0/3149609600 conn(0x7fa904069360 msgr2=0x7fa9040ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 1 -- 172.21.15.121:0/3149609600 shutdown_connections 2024-09-13T13:52:15.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.149+0000 7fa90a9d5640 1 -- 172.21.15.121:0/3149609600 wait complete. 2024-09-13T13:52:15.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.150+0000 7fa90a9d5640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.150+0000 7fa90a9d5640 10 monclient: monmap: 2024-09-13T13:52:15.159 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.160 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.151+0000 7fa90a9d5640 1 Processor -- start 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.151+0000 7fa90a9d5640 1 -- start start 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.151+0000 7fa90a9d5640 10 monclient: init 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient: _add_conns ranks=[8,7,1,4,5,0,3,2,6] 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient(hunting): picked mon.noname-e con 0x7fa8e4007620 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient(hunting): picked mon.noname-i con 0x7fa90409edd0 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient(hunting): picked mon.noname-f con 0x7fa90409f1f0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9040a1390 con 0x7fa90409f1f0 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9040a5c40 con 0x7fa90409edd0 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9041360c0 con 0x7fa8e4007620 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa90a9d5640 10 monclient(hunting): authenticate will time out at 814.781404s 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa903fff640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fa8e4007620 0x7fa90409e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:50546/0 (socket says 172.21.15.121:50546) 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa903fff640 1 -- 172.21.15.121:0/38859178 learned_addr learned my addr 172.21.15.121:0/38859178 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa9037fe640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.153:6792/0 conn(0x7fa90409edd0 legacy=0x7fa9041d2e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.152+0000 7fa908f72640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.153:6789/0 conn(0x7fa90409f1f0 legacy=0x7fa9041d65c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.154+0000 7fa903fff640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.121:6793/0 conn(0x7fa8e4007620 legacy=0x7fa90409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3851918495 0 0) 0x7fa9041360c0 con 0x7fa8e4007620 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): my global_id is 4517 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e40091e0 con 0x7fa8e4007620 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3795110769 0 0) 0x7fa8e40091e0 con 0x7fa8e4007620 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa9041360c0 con 0x7fa8e4007620 2024-09-13T13:52:15.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8f4003610 con 0x7fa8e4007620 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): dump: 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.163 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 180023058 0 0) 0x7fa9041360c0 con 0x7fa8e4007620 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.153:6792/0 conn(0x7fa90409edd0 legacy=0x7fa9041d2e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.153:6789/0 conn(0x7fa90409f1f0 legacy=0x7fa9041d65c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 monclient: found mon.e 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9040a10b0 con 0x7fa8e4007620 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa900ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa90a9d5640 5 monclient: authenticate success, global_id 4517 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa90a9d5640 10 monclient: _renew_subs 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa90a9d5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.155+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa90409f910 con 0x7fa8e4007620 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa90a9d5640 10 monclient: _renew_subs 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa90a9d5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa904135a60 con 0x7fa8e4007620 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8f4003b80 con 0x7fa8e4007620 2024-09-13T13:52:15.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa900ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8f40040b0 con 0x7fa8e4007620 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa900ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa900ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa900ff9640 10 monclient: dump: 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.166 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.156+0000 7fa901ffb640 4 set_mon_vals no callback set 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fa8f4004300 con 0x7fa8e4007620 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa90a9d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa90a9d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa90a9d5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90409d470 con 0x7fa8e4007620 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fa8f404e260 con 0x7fa8e4007620 2024-09-13T13:52:15.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.157+0000 7fa9037fe640 10 monclient: get_auth_request con 0x7fa8e4040eb0 auth_method 0 2024-09-13T13:52:15.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.161+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fa8f401b9a0 con 0x7fa8e4007620 2024-09-13T13:52:15.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.161+0000 7fa900ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.161+0000 7fa900ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa90a9d5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa90a9d5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa90a9d5640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa904074b90 con 0x7fa8e4007620 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa900ff9640 1 -- 172.21.15.121:0/38859178 <== mon.8 v1:172.21.15.121:6793/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2732 (unknown 254746657 0 2113172806) 0x7fa8f4020450 con 0x7fa8e4007620 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa900ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T13:52:15.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.411+0000 7fa900ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:15.413 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":22,"quorum":[0,2,3,4,5,6,8],"quorum_names":["a","b","g","c","h","d","e"],"quorum_leader_name":"h","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","modified":"2024-09-13T13:50:47.391384Z","created":"2024-09-13T13:50:47.391384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6791","nonce":0}]},"addr":"172.21.15.121:6791/0","public_addr":"172.21.15.121:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6791","nonce":0}]},"addr":"172.21.15.153:6791/0","public_addr":"172.21.15.153:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6792","nonce":0}]},"addr":"172.21.15.121:6792/0","public_addr":"172.21.15.121:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6792","nonce":0}]},"addr":"172.21.15.153:6792/0","public_addr":"172.21.15.153:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6793","nonce":0}]},"addr":"172.21.15.121:6793/0","public_addr":"172.21.15.121:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T13:52:15.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.153:6812/1206688214 conn(0x7fa8e4040eb0 legacy=0x7fa8e4043370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 10 monclient: shutdown 2024-09-13T13:52:15.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 >> v1:172.21.15.121:6793/0 conn(0x7fa8e4007620 legacy=0x7fa90409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 shutdown_connections 2024-09-13T13:52:15.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 >> 172.21.15.121:0/38859178 conn(0x7fa904069360 msgr2=0x7fa9040a8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 shutdown_connections 2024-09-13T13:52:15.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.413+0000 7fa90a9d5640 1 -- 172.21.15.121:0/38859178 wait complete. 2024-09-13T13:52:15.425 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-09-13T13:52:15.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-09-13T13:52:15.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T13:52:15.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.521+0000 7f1eda423640 10 monclient: get_monmap_and_config 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.521+0000 7f1eda423640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.521+0000 7f1eda423640 10 monclient: monmap: 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:15.522604+0000 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:15.522604+0000 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.523 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.524 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.524 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.522+0000 7f1eda423640 1 Processor -- start 2024-09-13T13:52:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.522+0000 7f1eda423640 1 -- start start 2024-09-13T13:52:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.522+0000 7f1eda423640 10 monclient: init 2024-09-13T13:52:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient: _add_conns ranks=[1,2,6,4,0,3,5,8,7] 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient(hunting): picked mon.noname-f con 0x7f1ed4138bd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient(hunting): picked mon.noname-b con 0x7f1ed4138fb0 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient(hunting): picked mon.noname-d con 0x7f1ed4139420 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed409e100 con 0x7f1ed4138bd0 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed409db40 con 0x7f1ed4138fb0 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed409de20 con 0x7f1ed4139420 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1eda423640 10 monclient(hunting): authenticate will time out at 815.152262s 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1ed37fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f1ed4138fb0 0x7f1ed413df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:43394/0 (socket says 172.21.15.121:43394) 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1ed37fe640 1 -- 172.21.15.121:0/752546672 learned_addr learned my addr 172.21.15.121:0/752546672 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.523+0000 7f1ed3fff640 1 -- 172.21.15.121:0/752546672 >> v1:172.21.15.153:6789/0 conn(0x7f1ed4138bd0 legacy=0x7f1ed40a5690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed89c0640 1 -- 172.21.15.121:0/752546672 >> v1:172.21.15.121:6792/0 conn(0x7f1ed4139420 legacy=0x7f1ed4141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed37fe640 1 -- 172.21.15.121:0/752546672 >> v1:172.21.15.121:6790/0 conn(0x7f1ed4138fb0 legacy=0x7f1ed413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983255837 0 0) 0x7f1ed409db40 con 0x7f1ed4138fb0 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 4466 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ec0003520 con 0x7f1ed4138fb0 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2742220636 0 0) 0x7f1ed409de20 con 0x7f1ed4139420 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient(hunting): my global_id is 4515 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed409db40 con 0x7f1ed4139420 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4253870314 0 0) 0x7f1ed409db40 con 0x7f1ed4139420 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 >> v1:172.21.15.121:6790/0 conn(0x7f1ed4138fb0 legacy=0x7f1ed413df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 >> v1:172.21.15.153:6789/0 conn(0x7f1ed4138bd0 legacy=0x7f1ed40a5690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 monclient: found mon.noname-d 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed40a5da0 con 0x7f1ed4139420 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ecc001e10 con 0x7f1ed4139420 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1eda423640 5 monclient: authenticate success, global_id 4515 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1eda423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.525+0000 7f1ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 10 monclient: dump: 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.533 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ecc003990 con 0x7f1ed4139420 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 1 -- 172.21.15.121:0/752546672 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ecc003f00 con 0x7f1ed4139420 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 4 set_mon_vals no callback set 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1ed2ffd640 10 monclient: dump: 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.535 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 10 monclient: shutdown 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 1 -- 172.21.15.121:0/752546672 >> v1:172.21.15.121:6792/0 conn(0x7f1ed4139420 legacy=0x7f1ed4141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 1 -- 172.21.15.121:0/752546672 shutdown_connections 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 1 -- 172.21.15.121:0/752546672 >> 172.21.15.121:0/752546672 conn(0x7f1ed40ab800 msgr2=0x7f1ed40abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 1 -- 172.21.15.121:0/752546672 shutdown_connections 2024-09-13T13:52:15.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.526+0000 7f1eda423640 1 -- 172.21.15.121:0/752546672 wait complete. 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.527+0000 7f1eda423640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.527+0000 7f1eda423640 10 monclient: monmap: 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.537 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.527+0000 7f1eda423640 1 Processor -- start 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.527+0000 7f1eda423640 1 -- start start 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.527+0000 7f1eda423640 10 monclient: init 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient: _add_conns ranks=[6,1,8,0,4,3,2,7,5] 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient(hunting): picked mon.noname-d con 0x7f1ed4138bd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient(hunting): picked mon.noname-f con 0x7f1ed4138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient(hunting): picked mon.noname-e con 0x7f1ed4139420 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed41303a0 con 0x7f1ed4138fb0 2024-09-13T13:52:15.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed4131e60 con 0x7f1ed4138bd0 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed4132140 con 0x7f1ed4139420 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1eda423640 10 monclient(hunting): authenticate will time out at 815.157203s 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1ed37fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f1ed4138fb0 legacy=0x7f1ed412efc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1ed3fff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f1ed4138bd0 0x7f1ed412e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:48020/0 (socket says 172.21.15.121:48020) 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1ed3fff640 1 -- 172.21.15.121:0/15691179 learned_addr learned my addr 172.21.15.121:0/15691179 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.528+0000 7f1ed89c0640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f1ed4139420 0x7f1ed41d5fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:50552/0 (socket says 172.21.15.121:50552) 2024-09-13T13:52:15.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed3fff640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6792/0 conn(0x7f1ed4138bd0 legacy=0x7f1ed412e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed89c0640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6793/0 conn(0x7f1ed4139420 legacy=0x7f1ed41d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25867214 0 0) 0x7f1ed4132140 con 0x7f1ed4139420 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): my global_id is 4526 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb4003520 con 0x7f1ed4139420 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257258761 0 0) 0x7f1ed4131e60 con 0x7f1ed4138bd0 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): my global_id is 4524 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed4132140 con 0x7f1ed4138bd0 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1973231583 0 0) 0x7f1eb4003520 con 0x7f1ed4139420 2024-09-13T13:52:15.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ed4131e60 con 0x7f1ed4139420 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ecc003e80 con 0x7f1ed4139420 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): dump: 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.546 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1321207170 0 0) 0x7f1ed4132140 con 0x7f1ed4138bd0 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eb4003520 con 0x7f1ed4138bd0 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec4004660 con 0x7f1ed4138bd0 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:15.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 10 monclient(hunting): dump: 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.548 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.530+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1729125587 0 0) 0x7f1ed4131e60 con 0x7f1ed4139420 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6792/0 conn(0x7f1ed4138bd0 legacy=0x7f1ed412e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.153:6789/0 conn(0x7f1ed4138fb0 legacy=0x7f1ed412efc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 monclient: found mon.e 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed412f6d0 con 0x7f1ed4139420 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 5 monclient: authenticate success, global_id 4526 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 10 monclient: _renew_subs 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed412f960 con 0x7f1ed4139420 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 10 monclient: _renew_subs 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ed41310e0 con 0x7f1ed4139420 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ecc001db0 con 0x7f1ed4139420 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ecc004610 con 0x7f1ed4139420 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed27fc640 4 set_mon_vals no callback set 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 10 monclient: dump: 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.551 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.531+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1ecc012020 con 0x7f1ed4139420 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.532+0000 7f1eda423640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.532+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1eb80065c0 con 0x7f1eb8002860 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.532+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f1ecc04e390 con 0x7f1ed4139420 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.532+0000 7f1ed37fe640 10 monclient: get_auth_request con 0x7f1eb40400d0 auth_method 0 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed3fff640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6789/0 conn(0x7f1eb8002860 legacy=0x7f1eb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3390133853 0 0) 0x7f1eb80065c0 con 0x7f1eb8002860 2024-09-13T13:52:15.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient(hunting): my global_id is 4545 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed4131e60 con 0x7f1eb8002860 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3991978646 0 0) 0x7f1ed4131e60 con 0x7f1eb8002860 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eb80065c0 con 0x7f1eb8002860 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec4004fe0 con 0x7f1eb8002860 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 10 monclient: dump: 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.553 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.554 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 292071446 0 0) 0x7f1eb80065c0 con 0x7f1eb8002860 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.534+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1eb800a160 con 0x7f1eb8002860 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.535+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1ec40072c0 con 0x7f1eb8002860 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.535+0000 7f1ed0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.535+0000 7f1ed0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.535+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6789/0 conn(0x7f1eb8002860 legacy=0x7f1eb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.535+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6789/0 conn(0x7f1eb8002860 legacy=0x7f1eb8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.541+0000 7f1eda423640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.541+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1eb8009e50 con 0x7f1eb8009110 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed3fff640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6789/0 conn(0x7f1eb8009110 legacy=0x7f1eb80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3022719369 0 0) 0x7f1eb8009e50 con 0x7f1eb8009110 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient(hunting): my global_id is 4554 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eb80065c0 con 0x7f1eb8009110 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 219390874 0 0) 0x7f1eb80065c0 con 0x7f1eb8009110 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eb8009e50 con 0x7f1eb8009110 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1ec4005a00 con 0x7f1eb8009110 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient: dump: 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.556 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2348598580 0 0) 0x7f1eb8009e50 con 0x7f1eb8009110 2024-09-13T13:52:15.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 --> v1:172.21.15.121:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1eb800c3f0 con 0x7f1eb8009110 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 <== mon.0 v1:172.21.15.121:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7703 (unknown 0 0 368428124) 0x7f1ec4006e70 con 0x7f1eb8009110 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.543+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6789/0 conn(0x7f1eb8009110 legacy=0x7f1eb80095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1ed0ff9640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6789/0 conn(0x7f1eb8009110 legacy=0x7f1eb80095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:15.558 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 59201, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:15.559 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.560 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:15.562 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.563 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.564 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.565 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:15.566 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.567 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.568 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:15.570 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.571 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.572 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.573 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:15.574 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.575 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.576 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.577 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:15.578 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:15.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1eb40400d0 legacy=0x7f1eb4042590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 10 monclient: shutdown 2024-09-13T13:52:15.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 >> v1:172.21.15.121:6793/0 conn(0x7f1ed4139420 legacy=0x7f1ed41d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1ed89c0640 1 -- 172.21.15.121:0/15691179 reap_dead start 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1ed89c0640 1 -- 172.21.15.121:0/15691179 reap_dead start 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 shutdown_connections 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 >> 172.21.15.121:0/15691179 conn(0x7f1ed40ab800 msgr2=0x7f1ed412c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 shutdown_connections 2024-09-13T13:52:15.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.544+0000 7f1eda423640 1 -- 172.21.15.121:0/15691179 wait complete. 2024-09-13T13:52:15.580 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:15.580 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-09-13T13:52:15.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T13:52:15.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.678+0000 7feddd695640 10 monclient: get_monmap_and_config 2024-09-13T13:52:15.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.678+0000 7feddd695640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.678+0000 7feddd695640 10 monclient: monmap: 2024-09-13T13:52:15.679 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.679 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.679 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:15.679236+0000 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:15.679236+0000 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.680 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 1 Processor -- start 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 1 -- start start 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient: init 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient: _add_conns ranks=[7,3,1,4,2,5,0,8,6] 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient(hunting): picked mon.noname-i con 0x7fedd80a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient(hunting): picked mon.noname-g con 0x7fedd809d150 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:15.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient(hunting): picked mon.noname-f con 0x7fedd80a6c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd809e210 con 0x7fedd80a6c50 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd80a7ef0 con 0x7fedd809d150 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd809d970 con 0x7fedd80a6160 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7feddd695640 10 monclient(hunting): authenticate will time out at 815.308752s 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7fedd77fe640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fedd80a6c50 legacy=0x7fedd8141aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7fedd6ffd640 1 -- >> v1:172.21.15.153:6792/0 conn(0x7fedd80a6160 legacy=0x7fedd80a6540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.679+0000 7fedd67fc640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fedd809d150 0x7fedd813e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:41782/0 (socket says 172.21.15.121:41782) 2024-09-13T13:52:15.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.680+0000 7fedd67fc640 1 -- 172.21.15.121:0/3443775254 learned_addr learned my addr 172.21.15.121:0/3443775254 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd67fc640 1 -- 172.21.15.121:0/3443775254 >> v1:172.21.15.153:6790/0 conn(0x7fedd809d150 legacy=0x7fedd813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3581386066 0 0) 0x7fedd80a7ef0 con 0x7fedd809d150 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient(hunting): my global_id is 4440 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc0003520 con 0x7fedd809d150 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4101887424 0 0) 0x7fedc0003520 con 0x7fedd809d150 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 >> v1:172.21.15.153:6792/0 conn(0x7fedd80a6160 legacy=0x7fedd80a6540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 >> v1:172.21.15.153:6789/0 conn(0x7fedd80a6c50 legacy=0x7fedd8141aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:15.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 monclient: found mon.noname-g 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: _send_mon_message to mon.noname-g at v1:172.21.15.153:6790/0 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 --> v1:172.21.15.153:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd809e630 con 0x7fedd809d150 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7feddd695640 5 monclient: authenticate success, global_id 4440 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedcc001ae0 con 0x7fedd809d150 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7feddd695640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-13T13:52:15.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7fedd5ffb640 10 monclient: dump: 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.689 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.682+0000 7feddd695640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 <== mon.3 v1:172.21.15.153:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedcc0048a0 con 0x7fedd809d150 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7fedd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7fedd5ffb640 1 -- 172.21.15.121:0/3443775254 <== mon.3 v1:172.21.15.153:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedcc004e10 con 0x7fedd809d150 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 4 set_mon_vals no callback set 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7fedd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7fedd5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7fedd5ffb640 10 monclient: dump: 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.690 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.691 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 10 monclient: shutdown 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 1 -- 172.21.15.121:0/3443775254 >> v1:172.21.15.153:6790/0 conn(0x7fedd809d150 legacy=0x7fedd813e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 1 -- 172.21.15.121:0/3443775254 shutdown_connections 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 1 -- 172.21.15.121:0/3443775254 >> 172.21.15.121:0/3443775254 conn(0x7fedd80ab940 msgr2=0x7fedd80abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 1 -- 172.21.15.121:0/3443775254 shutdown_connections 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.683+0000 7feddd695640 1 -- 172.21.15.121:0/3443775254 wait complete. 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.684+0000 7feddd695640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.684+0000 7feddd695640 10 monclient: monmap: 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:15.692 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.684+0000 7feddd695640 1 Processor -- start 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.684+0000 7feddd695640 1 -- start start 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.684+0000 7feddd695640 10 monclient: init 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient: _add_conns ranks=[1,4,6,7,8,2,5,3,0] 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient(hunting): picked mon.noname-f con 0x7fedd809d150 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient(hunting): picked mon.noname-c con 0x7fedd80a6160 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient(hunting): picked mon.noname-d con 0x7fedd80a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd8124870 con 0x7fedd809d150 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd80a8f10 con 0x7fedd80a6160 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd81268f0 con 0x7fedd80a6c50 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7feddd695640 10 monclient(hunting): authenticate will time out at 815.314222s 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7fedd6ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7fedd809d150 legacy=0x7fedd81391f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7fedd67fc640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fedd80a6160 0x7fedd81d2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:48672/0 (socket says 172.21.15.121:48672) 2024-09-13T13:52:15.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7fedd77fe640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fedd80a6c50 0x7fedd81d62a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:48022/0 (socket says 172.21.15.121:48022) 2024-09-13T13:52:15.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.685+0000 7fedd67fc640 1 -- 172.21.15.121:0/164744509 learned_addr learned my addr 172.21.15.121:0/164744509 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedd77fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6792/0 conn(0x7fedd80a6c50 legacy=0x7fedd81d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedd67fc640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6791/0 conn(0x7fedd80a6160 legacy=0x7fedd81d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 114314881 0 0) 0x7fedd81268f0 con 0x7fedd80a6c50 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): my global_id is 4533 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedc4003520 con 0x7fedd80a6c50 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1802520284 0 0) 0x7fedd809e630 con 0x7fedd80a6160 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): my global_id is 4459 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedd81268f0 con 0x7fedd80a6160 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2657824529 0 0) 0x7fedc4003520 con 0x7fedd80a6c50 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedd809e630 con 0x7fedd80a6c50 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3558140946 0 0) 0x7fedd81268f0 con 0x7fedd80a6160 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedc4003520 con 0x7fedd80a6160 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedd0004840 con 0x7fedd80a6c50 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): dump: 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.701 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedcc004d90 con 0x7fedd80a6160 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.687+0000 7fedbf7fe640 10 monclient(hunting): dump: 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.702 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 359171594 0 0) 0x7fedd809e630 con 0x7fedd80a6c50 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6791/0 conn(0x7fedd80a6160 legacy=0x7fedd81d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.153:6789/0 conn(0x7fedd809d150 legacy=0x7fedd81391f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 monclient: found mon.d 2024-09-13T13:52:15.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd8126bd0 con 0x7fedd80a6c50 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 5 monclient: authenticate success, global_id 4533 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 10 monclient: _renew_subs 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fedd80a40a0 con 0x7fedd80a6c50 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 10 monclient: _renew_subs 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fedd81cda90 con 0x7fedd80a6c50 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fedd0004b10 con 0x7fedd80a6c50 2024-09-13T13:52:15.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedd0005080 con 0x7fedd80a6c50 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedd57fa640 4 set_mon_vals no callback set 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7fedbf7fe640 10 monclient: dump: 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.705 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.688+0000 7feddd695640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.689+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fedb80065c0 con 0x7fedb8002860 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.689+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fedd0005220 con 0x7fedd80a6c50 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.689+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fedd004e4c0 con 0x7fedd80a6c50 2024-09-13T13:52:15.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.689+0000 7fedd67fc640 10 monclient: get_auth_request con 0x7fedc403fed0 auth_method 0 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedd6ffd640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6790/0 conn(0x7fedb8002860 legacy=0x7fedb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854184661 0 0) 0x7fedb80065c0 con 0x7fedb8002860 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient(hunting): my global_id is 4475 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedd809e630 con 0x7fedb8002860 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1900836213 0 0) 0x7fedd809e630 con 0x7fedb8002860 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedb80065c0 con 0x7fedb8002860 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedc80043c0 con 0x7fedb8002860 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 10 monclient: dump: 2024-09-13T13:52:15.707 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.708 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 142972496 0 0) 0x7fedb80065c0 con 0x7fedb8002860 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.691+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fedb800a160 con 0x7fedb8002860 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.692+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fedc8006770 con 0x7fedb8002860 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.692+0000 7fedbf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.692+0000 7fedbf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.692+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6790/0 conn(0x7fedb8002860 legacy=0x7fedb8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.692+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6790/0 conn(0x7fedb8002860 legacy=0x7fedb8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.698+0000 7feddd695640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:15.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.698+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fedb8009e70 con 0x7fedb8009130 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedd6ffd640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6790/0 conn(0x7fedb8009130 legacy=0x7fedb80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 913649358 0 0) 0x7fedb8009e70 con 0x7fedb8009130 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient(hunting): my global_id is 4484 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedb80065c0 con 0x7fedb8009130 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3729322124 0 0) 0x7fedb80065c0 con 0x7fedb8009130 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fedb8009e70 con 0x7fedb8009130 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fedc8004df0 con 0x7fedb8009130 2024-09-13T13:52:15.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 10 monclient: dump: 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.712 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1324961820 0 0) 0x7fedb8009e70 con 0x7fedb8009130 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.700+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 --> v1:172.21.15.121:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fedb800c410 con 0x7fedb8009130 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 <== mon.2 v1:172.21.15.121:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7703 (unknown 0 0 787948858) 0x7fedc80061d0 con 0x7fedb8009130 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedbf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedbf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6790/0 conn(0x7fedb8009130 legacy=0x7fedb80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedbf7fe640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6790/0 conn(0x7fedb8009130 legacy=0x7fedb80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 59368, 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:15.714 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.715 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.716 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:15.717 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:15.718 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.719 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:15.720 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.721 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.722 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:15.723 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.724 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.725 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.726 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.727 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.729 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:15.730 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:52:15.732 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.733 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:15.734 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.153:6812/1206688214 conn(0x7fedc403fed0 legacy=0x7fedc4042390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 10 monclient: shutdown 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 >> v1:172.21.15.121:6792/0 conn(0x7fedd80a6c50 legacy=0x7fedd81d62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedd77fe640 1 -- 172.21.15.121:0/164744509 reap_dead start 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7fedd77fe640 1 -- 172.21.15.121:0/164744509 reap_dead start 2024-09-13T13:52:15.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 shutdown_connections 2024-09-13T13:52:15.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 >> 172.21.15.121:0/164744509 conn(0x7fedd80ab940 msgr2=0x7fedd81c1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 shutdown_connections 2024-09-13T13:52:15.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.701+0000 7feddd695640 1 -- 172.21.15.121:0/164744509 wait complete. 2024-09-13T13:52:15.736 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:15.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-09-13T13:52:15.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.835+0000 7f54201d1640 10 monclient: get_monmap_and_config 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.835+0000 7f54201d1640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.835+0000 7f54201d1640 10 monclient: monmap: 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:15.836246+0000 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:15.836246+0000 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.837 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 1 Processor -- start 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 1 -- start start 2024-09-13T13:52:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient: init 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient: _add_conns ranks=[1,0,5,2,7,4,3,6,8] 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient(hunting): picked mon.noname-f con 0x7f5418138bd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient(hunting): picked mon.noname-a con 0x7f5418138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient(hunting): picked mon.noname-h con 0x7f5418139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541809e100 con 0x7f5418138fb0 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541809db40 con 0x7f5418138bd0 2024-09-13T13:52:15.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541809de20 con 0x7f5418139420 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f54201d1640 10 monclient(hunting): authenticate will time out at 815.465920s 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f541d76c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5418138fb0 0x7f541813df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38396/0 (socket says 172.21.15.121:38396) 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.836+0000 7f541df6d640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f5418138bd0 legacy=0x7f54180a5690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.837+0000 7f541d76c640 1 -- 172.21.15.121:0/2666450985 learned_addr learned my addr 172.21.15.121:0/2666450985 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.837+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 DispatchQueue::entry inject delay of 0.002000 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541d76c640 1 -- 172.21.15.121:0/2666450985 >> v1:172.21.15.121:6789/0 conn(0x7f5418138fb0 legacy=0x7f541813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1169513955 0 0) 0x7f541809e100 con 0x7f5418138fb0 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient(hunting): my global_id is 4563 2024-09-13T13:52:15.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5404003520 con 0x7f5418138fb0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541e76e640 1 -- 172.21.15.121:0/2666450985 >> v1:172.21.15.153:6791/0 conn(0x7f5418139420 legacy=0x7f5418141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3729287629 0 0) 0x7f5404003520 con 0x7f5418138fb0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 >> v1:172.21.15.153:6791/0 conn(0x7f5418139420 legacy=0x7f5418141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 >> v1:172.21.15.153:6789/0 conn(0x7f5418138bd0 legacy=0x7f54180a5690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 monclient: found mon.noname-a 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54180a5da0 con 0x7f5418138fb0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5408001ae0 con 0x7f5418138fb0 2024-09-13T13:52:15.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f54201d1640 5 monclient: authenticate success, global_id 4563 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f54201d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: dump: 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.846 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f54201d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f54080048a0 con 0x7f5418138fb0 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 1 -- 172.21.15.121:0/2666450985 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5408004e10 con 0x7f5418138fb0 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f54201d1640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f54201d1640 4 set_mon_vals no callback set 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.839+0000 7f541cf6b640 10 monclient: dump: 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.848 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 10 monclient: shutdown 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 1 -- 172.21.15.121:0/2666450985 >> v1:172.21.15.121:6789/0 conn(0x7f5418138fb0 legacy=0x7f541813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 1 -- 172.21.15.121:0/2666450985 shutdown_connections 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 1 -- 172.21.15.121:0/2666450985 >> 172.21.15.121:0/2666450985 conn(0x7f54180ab800 msgr2=0x7f54180abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 1 -- 172.21.15.121:0/2666450985 shutdown_connections 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 1 -- 172.21.15.121:0/2666450985 wait complete. 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.840+0000 7f54201d1640 10 monclient: monmap: 2024-09-13T13:52:15.849 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.850 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 1 Processor -- start 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 1 -- start start 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient: init 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient: _add_conns ranks=[0,1,8,7,3,4,5,6,2] 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient(hunting): picked mon.noname-a con 0x7f5418138bd0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient(hunting): picked mon.noname-f con 0x7f5418138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient(hunting): picked mon.noname-e con 0x7f5418139420 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54181301a0 con 0x7f5418138bd0 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5418131c60 con 0x7f5418138fb0 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5418131f40 con 0x7f5418139420 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f54201d1640 10 monclient(hunting): authenticate will time out at 815.470625s 2024-09-13T13:52:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f541df6d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5418138bd0 0x7f541812e920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38406/0 (socket says 172.21.15.121:38406) 2024-09-13T13:52:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f541df6d640 1 -- 172.21.15.121:0/2420166974 learned_addr learned my addr 172.21.15.121:0/2420166974 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.841+0000 7f541d76c640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6789/0 conn(0x7f5418138fb0 legacy=0x7f541812f030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f541df6d640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6789/0 conn(0x7f5418138bd0 legacy=0x7f541812e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f541e76e640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6793/0 conn(0x7f5418139420 legacy=0x7f541812f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1706619444 0 0) 0x7f54181301a0 con 0x7f5418138bd0 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): my global_id is 4572 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53fc003520 con 0x7f5418138bd0 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367704748 0 0) 0x7f5418131f40 con 0x7f5418139420 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 10 monclient(hunting): my global_id is 4535 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.843+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54181301a0 con 0x7f5418139420 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3252666288 0 0) 0x7f53fc003520 con 0x7f5418138bd0 2024-09-13T13:52:15.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5418131f40 con 0x7f5418138bd0 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5414004670 con 0x7f5418138bd0 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.858 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2774752915 0 0) 0x7f54181301a0 con 0x7f5418139420 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53fc003520 con 0x7f5418139420 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5410002030 con 0x7f5418139420 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.859 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.860 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1095338753 0 0) 0x7f5418131f40 con 0x7f5418138bd0 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6793/0 conn(0x7f5418139420 legacy=0x7f541812f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6789/0 conn(0x7f5418138fb0 legacy=0x7f541812f030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 monclient: found mon.a 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5418132920 con 0x7f5418138bd0 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 5 monclient: authenticate success, global_id 4572 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 10 monclient: _renew_subs 2024-09-13T13:52:15.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5418132bb0 con 0x7f5418138bd0 2024-09-13T13:52:15.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 10 monclient: _renew_subs 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f54181d71c0 con 0x7f5418138bd0 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5414004940 con 0x7f5418138bd0 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5414004eb0 con 0x7f5418138bd0 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540ffff640 4 set_mon_vals no callback set 2024-09-13T13:52:15.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.844+0000 7f540e7fc640 10 monclient: dump: 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.865 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f54201d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f54000065c0 con 0x7f5400002860 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f5414005050 con 0x7f5418138bd0 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f541404e290 con 0x7f5418138bd0 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f541e76e640 10 monclient: get_auth_request con 0x7f53fc03fec0 auth_method 0 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f541e76e640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6812/1206688214 conn(0x7f53fc03fec0 legacy=0x7f53fc042380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f541e76e640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6812/1206688214 conn(0x7f53fc03fec0 legacy=0x7f53fc042380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f541e76e640 1 --1- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6812/1206688214 conn(0x7f53fc03fec0 0x7f53fc042380 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f541e76e640 1 --1- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6812/1206688214 conn(0x7f53fc03fec0 0x7f53fc042380 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.845+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.153:6812/1206688214 conn(0x7f53fc03fec0 legacy=0x7f53fc042380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f541d76c640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6791/0 conn(0x7f5400002860 legacy=0x7f5400004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718356340 0 0) 0x7f54000065c0 con 0x7f5400002860 2024-09-13T13:52:15.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient(hunting): my global_id is 4468 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5418131f40 con 0x7f5400002860 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 725510582 0 0) 0x7f5418131f40 con 0x7f5400002860 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54000065c0 con 0x7f5400002860 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f54080047e0 con 0x7f5400002860 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 10 monclient: dump: 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.867 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 602900240 0 0) 0x7f54000065c0 con 0x7f5400002860 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.847+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f540000a160 con 0x7f5400002860 2024-09-13T13:52:15.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.848+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f54080047e0 con 0x7f5400002860 2024-09-13T13:52:15.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.848+0000 7f540e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:15.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.848+0000 7f540e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:15.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.848+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6791/0 conn(0x7f5400002860 legacy=0x7f5400004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.848+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6791/0 conn(0x7f5400002860 legacy=0x7f5400004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.854+0000 7f54201d1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:15.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.854+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5400009e50 con 0x7f5400009110 2024-09-13T13:52:15.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f541d76c640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6791/0 conn(0x7f5400009110 legacy=0x7f54000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3391039183 0 0) 0x7f5400009e50 con 0x7f5400009110 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f540e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f540e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f540e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f540e7fc640 10 monclient(hunting): my global_id is 4477 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.856+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54000065c0 con 0x7f5400009110 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4043860766 0 0) 0x7f54000065c0 con 0x7f5400009110 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5400009e50 con 0x7f5400009110 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5408005150 con 0x7f5400009110 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 10 monclient: dump: 2024-09-13T13:52:15.871 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:15.872 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2974582028 0 0) 0x7f5400009e50 con 0x7f5400009110 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 --> v1:172.21.15.121:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f540000c3f0 con 0x7f5400009110 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 <== mon.4 v1:172.21.15.121:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7536 (unknown 0 0 2038861136) 0x7f5408007d80 con 0x7f5400009110 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6791/0 conn(0x7f5400009110 legacy=0x7f54000095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f540e7fc640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6791/0 conn(0x7f5400009110 legacy=0x7f54000095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f541e76e640 1 -- 172.21.15.121:0/2420166974 reap_dead start 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:15.873 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 59526, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:15.878 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.879 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.880 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.881 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.883 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:15.884 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.885 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.886 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:15.887 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.888 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.889 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:15.890 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:15.891 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:15.892 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:15.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 10 monclient: shutdown 2024-09-13T13:52:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 >> v1:172.21.15.121:6789/0 conn(0x7f5418138bd0 legacy=0x7f541812e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 shutdown_connections 2024-09-13T13:52:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 >> 172.21.15.121:0/2420166974 conn(0x7f54180ab800 msgr2=0x7f54180a6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 shutdown_connections 2024-09-13T13:52:15.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.857+0000 7f54201d1640 1 -- 172.21.15.121:0/2420166974 wait complete. 2024-09-13T13:52:15.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:15.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-09-13T13:52:15.895 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-13T13:52:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.992+0000 7f534d307640 10 monclient: get_monmap_and_config 2024-09-13T13:52:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.992+0000 7f534d307640 10 monclient: build_initial_monmap 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.992+0000 7f534d307640 10 monclient: monmap: 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:15.993654+0000 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:15.993654+0000 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:15.995 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.993+0000 7f534d307640 1 Processor -- start 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.993+0000 7f534d307640 1 -- start start 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.993+0000 7f534d307640 10 monclient: init 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.993+0000 7f534d307640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 10 monclient: _add_conns ranks=[5,7,8,6,2,1,0,3,4] 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 10 monclient(hunting): picked mon.noname-h con 0x7f53480a3e30 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 10 monclient(hunting): picked mon.noname-i con 0x7f53480a4210 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 10 monclient(hunting): picked mon.noname-e con 0x7f534813d050 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53480737f0 con 0x7f53480a3e30 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53480734a0 con 0x7f53480a4210 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53480a5b60 con 0x7f534813d050 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f534d307640 10 monclient(hunting): authenticate will time out at 815.623089s 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f53477fe640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f534813d050 0x7f534813f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:50574/0 (socket says 172.21.15.121:50574) 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f53477fe640 1 -- 172.21.15.121:0/1207735740 learned_addr learned my addr 172.21.15.121:0/1207735740 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.994+0000 7f53467fc640 1 -- >> v1:172.21.15.153:6792/0 conn(0x7f53480a4210 legacy=0x7f53480a4640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f53477fe640 1 -- 172.21.15.121:0/1207735740 >> v1:172.21.15.121:6793/0 conn(0x7f534813d050 legacy=0x7f534813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 569304583 0 0) 0x7f53480a5b60 con 0x7f534813d050 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): my global_id is 4544 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5346ffd640 1 -- 172.21.15.121:0/1207735740 >> v1:172.21.15.153:6791/0 conn(0x7f53480a3e30 legacy=0x7f5348073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5330003520 con 0x7f534813d050 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731693521 0 0) 0x7f53480737f0 con 0x7f53480a3e30 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient(hunting): my global_id is 4469 2024-09-13T13:52:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53480a5b60 con 0x7f53480a3e30 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 241418358 0 0) 0x7f5330003520 con 0x7f534813d050 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 >> v1:172.21.15.153:6792/0 conn(0x7f53480a4210 legacy=0x7f53480a4640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 >> v1:172.21.15.153:6791/0 conn(0x7f53480a3e30 legacy=0x7f5348073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 monclient: found mon.noname-e 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5348143b90 con 0x7f534813d050 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5340001e60 con 0x7f534813d050 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: dump: 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f534d307640 5 monclient: authenticate success, global_id 4544 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f534d307640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5340004c20 con 0x7f534813d050 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 1 -- 172.21.15.121:0/1207735740 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5340005190 con 0x7f534813d050 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f5345ffb640 10 monclient: dump: 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f534d307640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.996+0000 7f534d307640 4 set_mon_vals no callback set 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 10 monclient: shutdown 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 1 -- 172.21.15.121:0/1207735740 >> v1:172.21.15.121:6793/0 conn(0x7f534813d050 legacy=0x7f534813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 1 -- 172.21.15.121:0/1207735740 shutdown_connections 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 1 -- 172.21.15.121:0/1207735740 >> 172.21.15.121:0/1207735740 conn(0x7f53480a9430 msgr2=0x7f53480a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 1 -- 172.21.15.121:0/1207735740 shutdown_connections 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 1 -- 172.21.15.121:0/1207735740 wait complete. 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.997+0000 7f534d307640 10 monclient: monmap: 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 1 Processor -- start 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 1 -- start start 2024-09-13T13:52:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient: init 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient: _add_conns ranks=[5,2,1,7,3,0,6,8,4] 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient(hunting): picked mon.noname-h con 0x7f53480a3e30 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient(hunting): picked mon.noname-b con 0x7f53480a4210 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient(hunting): picked mon.noname-f con 0x7f534813d050 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5348120ec0 con 0x7f534813d050 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53481211a0 con 0x7f53480a4210 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53480a8750 con 0x7f53480a3e30 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.998+0000 7f534d307640 10 monclient(hunting): authenticate will time out at 815.627980s 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.999+0000 7f53467fc640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f53480a4210 0x7f53481d0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:43424/0 (socket says 172.21.15.121:43424) 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.999+0000 7f53467fc640 1 -- 172.21.15.121:0/2943517986 learned_addr learned my addr 172.21.15.121:0/2943517986 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:15.999+0000 7f53477fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.153:6789/0 conn(0x7f534813d050 legacy=0x7f53481d4410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:16.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f53467fc640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6790/0 conn(0x7f53480a4210 legacy=0x7f53481d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700628304 0 0) 0x7f53481211a0 con 0x7f53480a4210 2024-09-13T13:52:16.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): my global_id is 4493 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5334003520 con 0x7f53480a4210 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f5346ffd640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.153:6791/0 conn(0x7f53480a3e30 legacy=0x7f53481cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3408522242 0 0) 0x7f5334003520 con 0x7f53480a4210 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53481211a0 con 0x7f53480a4210 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f533c004670 con 0x7f53480a4210 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): dump: 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.013 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.014 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1795904498 0 0) 0x7f53480a8750 con 0x7f53480a3e30 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient(hunting): my global_id is 4478 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5334003520 con 0x7f53480a3e30 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1892609014 0 0) 0x7f53481211a0 con 0x7f53480a4210 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.153:6791/0 conn(0x7f53480a3e30 legacy=0x7f53481cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.153:6789/0 conn(0x7f534813d050 legacy=0x7f53481d4410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 monclient: found mon.b 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5348121480 con 0x7f53480a4210 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 5 monclient: authenticate success, global_id 4493 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 10 monclient: _renew_subs 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5348120be0 con 0x7f53480a4210 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 10 monclient: _renew_subs 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f53480a27c0 con 0x7f53480a4210 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f533c004980 con 0x7f53480a4210 2024-09-13T13:52:16.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f533c004ef0 con 0x7f53480a4210 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f53457fa640 4 set_mon_vals no callback set 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.001+0000 7f532f7fe640 10 monclient: dump: 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.017 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.002+0000 7f534d307640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.002+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f53280065c0 con 0x7f5328002860 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.002+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f533c005090 con 0x7f53480a4210 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.003+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f533c04e2d0 con 0x7f53480a4210 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.003+0000 7f53477fe640 10 monclient: get_auth_request con 0x7f533403fad0 auth_method 0 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f5346ffd640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6792/0 conn(0x7f5328002860 legacy=0x7f5328004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513832895 0 0) 0x7f53280065c0 con 0x7f5328002860 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f532f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f532f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f532f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f532f7fe640 10 monclient(hunting): my global_id is 4542 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.004+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53481211a0 con 0x7f5328002860 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2193771285 0 0) 0x7f53481211a0 con 0x7f5328002860 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53280065c0 con 0x7f5328002860 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5338001d20 con 0x7f5328002860 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 10 monclient: dump: 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.019 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2144453483 0 0) 0x7f53280065c0 con 0x7f5328002860 2024-09-13T13:52:16.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f532800a160 con 0x7f5328002860 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f53380057c0 con 0x7f5328002860 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6792/0 conn(0x7f5328002860 legacy=0x7f5328004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.005+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6792/0 conn(0x7f5328002860 legacy=0x7f5328004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.011+0000 7f534d307640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:16.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.011+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5328009e50 con 0x7f5328009110 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.013+0000 7f5346ffd640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6792/0 conn(0x7f5328009110 legacy=0x7f53280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74766396 0 0) 0x7f5328009e50 con 0x7f5328009110 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient(hunting): my global_id is 4551 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53280065c0 con 0x7f5328009110 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1217354361 0 0) 0x7f53280065c0 con 0x7f5328009110 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5328009e50 con 0x7f5328009110 2024-09-13T13:52:16.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5338003dc0 con 0x7f5328009110 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient: dump: 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.024 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3731288225 0 0) 0x7f5328009e50 con 0x7f5328009110 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 --> v1:172.21.15.121:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f532800c3a0 con 0x7f5328009110 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 <== mon.6 v1:172.21.15.121:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7536 (unknown 0 0 3706055476) 0x7f5338004060 con 0x7f5328009110 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6792/0 conn(0x7f5328009110 legacy=0x7f53280095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.014+0000 7f532f7fe640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6792/0 conn(0x7f5328009110 legacy=0x7f53280095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 59687, 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:16.026 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 0, 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.027 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.028 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:16.029 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:16.030 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.031 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:16.032 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.033 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.034 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.035 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.038 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.039 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.040 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:16.042 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.043 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.044 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:16.045 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.153:6812/1206688214 conn(0x7f533403fad0 legacy=0x7f5334041f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 10 monclient: shutdown 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 >> v1:172.21.15.121:6790/0 conn(0x7f53480a4210 legacy=0x7f53481d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f53477fe640 1 -- 172.21.15.121:0/2943517986 reap_dead start 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f53477fe640 1 -- 172.21.15.121:0/2943517986 reap_dead start 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 shutdown_connections 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 >> 172.21.15.121:0/2943517986 conn(0x7f53480a9430 msgr2=0x7f5348141b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 shutdown_connections 2024-09-13T13:52:16.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.015+0000 7f534d307640 1 -- 172.21.15.121:0/2943517986 wait complete. 2024-09-13T13:52:16.047 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:16.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-09-13T13:52:16.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.145+0000 7f9824448640 10 monclient: get_monmap_and_config 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.145+0000 7f9824448640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.145+0000 7f9824448640 10 monclient: monmap: 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:16.145932+0000 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:16.145932+0000 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.146 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.147 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.145+0000 7f9824448640 1 Processor -- start 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.145+0000 7f9824448640 1 -- start start 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.145+0000 7f9824448640 10 monclient: init 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient: _add_conns ranks=[6,0,1,2,8,7,4,5,3] 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient(hunting): picked mon.noname-d con 0x7f981c0a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient(hunting): picked mon.noname-a con 0x7f981c09d150 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient(hunting): picked mon.noname-f con 0x7f981c0a6c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981c09e210 con 0x7f981c09d150 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981c0a7ef0 con 0x7f981c0a6c50 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981c09d970 con 0x7f981c0a6160 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f9824448640 10 monclient(hunting): authenticate will time out at 815.775633s 2024-09-13T13:52:16.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f98219e3640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f981c09d150 0x7f981c13e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38420/0 (socket says 172.21.15.121:38420) 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f98221e4640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f981c0a6160 0x7f981c0a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:48050/0 (socket says 172.21.15.121:48050) 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f98219e3640 1 -- 172.21.15.121:0/139659751 learned_addr learned my addr 172.21.15.121:0/139659751 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.146+0000 7f98229e5640 1 -- 172.21.15.121:0/139659751 >> v1:172.21.15.153:6789/0 conn(0x7f981c0a6c50 legacy=0x7f981c141aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.148+0000 7f98221e4640 1 -- 172.21.15.121:0/139659751 >> v1:172.21.15.121:6792/0 conn(0x7f981c0a6160 legacy=0x7f981c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.148+0000 7f98219e3640 1 -- 172.21.15.121:0/139659751 >> v1:172.21.15.121:6789/0 conn(0x7f981c09d150 legacy=0x7f981c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160868650 0 0) 0x7f981c09e210 con 0x7f981c09d150 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): my global_id is 4581 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9808003520 con 0x7f981c09d150 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951938981 0 0) 0x7f981c09d970 con 0x7f981c0a6160 2024-09-13T13:52:16.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient(hunting): my global_id is 4560 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f981c09e210 con 0x7f981c0a6160 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1150036891 0 0) 0x7f9808003520 con 0x7f981c09d150 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 >> v1:172.21.15.121:6792/0 conn(0x7f981c0a6160 legacy=0x7f981c0a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 >> v1:172.21.15.153:6789/0 conn(0x7f981c0a6c50 legacy=0x7f981c141aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 monclient: found mon.noname-a 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981c09e630 con 0x7f981c09d150 2024-09-13T13:52:16.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9818001ba0 con 0x7f981c09d150 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f9824448640 5 monclient: authenticate success, global_id 4581 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: dump: 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.155 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f9824448640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f98180048a0 con 0x7f981c09d150 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 1 -- 172.21.15.121:0/139659751 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9818001d40 con 0x7f981c09d150 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f9824448640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f9824448640 4 set_mon_vals no callback set 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.149+0000 7f98211e2640 10 monclient: dump: 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.157 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 10 monclient: shutdown 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 1 -- 172.21.15.121:0/139659751 >> v1:172.21.15.121:6789/0 conn(0x7f981c09d150 legacy=0x7f981c13e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 1 -- 172.21.15.121:0/139659751 shutdown_connections 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 1 -- 172.21.15.121:0/139659751 >> 172.21.15.121:0/139659751 conn(0x7f981c0ab940 msgr2=0x7f981c0abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 1 -- 172.21.15.121:0/139659751 shutdown_connections 2024-09-13T13:52:16.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 1 -- 172.21.15.121:0/139659751 wait complete. 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.150+0000 7f9824448640 10 monclient: monmap: 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.159 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 1 Processor -- start 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 1 -- start start 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient: init 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient: _add_conns ranks=[5,3,8,7,1,4,6,2,0] 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient(hunting): picked mon.noname-h con 0x7f981c09d150 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient(hunting): picked mon.noname-g con 0x7f981c0a6160 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient(hunting): picked mon.noname-e con 0x7f981c0a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:16.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981c1ce450 con 0x7f981c0a6160 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981c1247d0 con 0x7f981c09d150 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f981c0a4040 con 0x7f981c0a6c50 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f9824448640 10 monclient(hunting): authenticate will time out at 815.780647s 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f98229e5640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f981c0a6c50 0x7f981c1d5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:50582/0 (socket says 172.21.15.121:50582) 2024-09-13T13:52:16.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.151+0000 7f98229e5640 1 -- 172.21.15.121:0/755023508 learned_addr learned my addr 172.21.15.121:0/755023508 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f98229e5640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f981c0a6c50 legacy=0x7f981c1d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372237176 0 0) 0x7f9814004100 con 0x7f981c0a6c50 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f980effd640 10 monclient(hunting): my global_id is 4553 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.153+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9800003520 con 0x7f981c0a6c50 2024-09-13T13:52:16.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f98221e4640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.153:6791/0 conn(0x7f981c09d150 legacy=0x7f981c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f98219e3640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.153:6790/0 conn(0x7f981c0a6160 legacy=0x7f981c1d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3307773388 0 0) 0x7f9800003520 con 0x7f981c0a6c50 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9814004100 con 0x7f981c0a6c50 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9814004ae0 con 0x7f981c0a6c50 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): dump: 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.166 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4164175705 0 0) 0x7f981c1247d0 con 0x7f981c09d150 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): my global_id is 4487 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9800003520 con 0x7f981c09d150 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 500831909 0 0) 0x7f981c1ce450 con 0x7f981c0a6160 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient(hunting): my global_id is 4449 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f981c1247d0 con 0x7f981c0a6160 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 863351862 0 0) 0x7f9814004100 con 0x7f981c0a6c50 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.153:6791/0 conn(0x7f981c09d150 legacy=0x7f981c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.153:6790/0 conn(0x7f981c0a6160 legacy=0x7f981c1d2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 monclient: found mon.e 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981c0a8ed0 con 0x7f981c0a6c50 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 5 monclient: authenticate success, global_id 4553 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 10 monclient: _renew_subs 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f981c126850 con 0x7f981c0a6c50 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 10 monclient: _renew_subs 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:16.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f981c125330 con 0x7f981c0a6c50 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9814004df0 con 0x7f981c0a6c50 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9814005320 con 0x7f981c0a6c50 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f98209e1640 4 set_mon_vals no callback set 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.154+0000 7f980effd640 10 monclient: dump: 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.155+0000 7f9824448640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.155+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f98040065c0 con 0x7f9804002860 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.155+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f98140055c0 con 0x7f981c0a6c50 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.155+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f981404e780 con 0x7f981c0a6c50 2024-09-13T13:52:16.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.155+0000 7f98219e3640 10 monclient: get_auth_request con 0x7f980003fd10 auth_method 0 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f98221e4640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f9804002860 legacy=0x7f9804004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 951739151 0 0) 0x7f98040065c0 con 0x7f9804002860 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient(hunting): my global_id is 4562 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9814004100 con 0x7f9804002860 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 417322307 0 0) 0x7f9814004100 con 0x7f9804002860 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f98040065c0 con 0x7f9804002860 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9810001320 con 0x7f9804002860 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:16.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 10 monclient: dump: 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.173 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.157+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2381206769 0 0) 0x7f98040065c0 con 0x7f9804002860 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.158+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f980400a160 con 0x7f9804002860 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.158+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9810006820 con 0x7f9804002860 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.158+0000 7f980effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.158+0000 7f980effd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.158+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f9804002860 legacy=0x7f9804004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.158+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f9804002860 legacy=0x7f9804004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.164+0000 7f9824448640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:16.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.164+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9804009e50 con 0x7f9804009110 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f98221e4640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f9804009110 legacy=0x7f98040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3159745130 0 0) 0x7f9804009e50 con 0x7f9804009110 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 10 monclient(hunting): my global_id is 4571 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98040065c0 con 0x7f9804009110 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 78309252 0 0) 0x7f98040065c0 con 0x7f9804009110 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.166+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9804009e50 con 0x7f9804009110 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9810005080 con 0x7f9804009110 2024-09-13T13:52:16.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 10 monclient: dump: 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.177 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2060777793 0 0) 0x7f9804009e50 con 0x7f9804009110 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 --> v1:172.21.15.121:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f980400c3a0 con 0x7f9804009110 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 <== mon.8 v1:172.21.15.121:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7536 (unknown 0 0 3865535513) 0x7f9810006560 con 0x7f9804009110 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f9804009110 legacy=0x7f98040095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f980effd640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f9804009110 legacy=0x7f98040095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 59825, 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:16.179 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.180 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:16.182 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:16.183 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:16.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.185 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.186 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:16.187 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.188 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.190 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:16.191 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.192 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.193 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.194 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:16.195 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.197 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.198 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:16.198 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.198 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.198 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:16.198 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:16.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.153:6812/1206688214 conn(0x7f980003fd10 legacy=0x7f98000421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f9824448640 10 monclient: shutdown 2024-09-13T13:52:16.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f9824448640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 >> v1:172.21.15.121:6793/0 conn(0x7f981c0a6c50 legacy=0x7f981c1d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f98229e5640 1 -- 172.21.15.121:0/755023508 reap_dead start 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.167+0000 7f98229e5640 1 -- 172.21.15.121:0/755023508 reap_dead start 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.168+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 shutdown_connections 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.168+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 >> 172.21.15.121:0/755023508 conn(0x7f981c0ab940 msgr2=0x7f981c0aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.168+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 shutdown_connections 2024-09-13T13:52:16.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.168+0000 7f9824448640 1 -- 172.21.15.121:0/755023508 wait complete. 2024-09-13T13:52:16.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:16.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-09-13T13:52:16.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-13T13:52:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.298+0000 7f10ff7b4640 10 monclient: get_monmap_and_config 2024-09-13T13:52:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.298+0000 7f10ff7b4640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.298+0000 7f10ff7b4640 10 monclient: monmap: 2024-09-13T13:52:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.299 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:16.299258+0000 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:16.299258+0000 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.300 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 1 Processor -- start 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 1 -- start start 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient: init 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient: _add_conns ranks=[3,1,5,7,0,2,4,8,6] 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient(hunting): picked mon.noname-g con 0x7f10f8138bd0 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient(hunting): picked mon.noname-f con 0x7f10f8138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:16.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient(hunting): picked mon.noname-h con 0x7f10f8139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f809e100 con 0x7f10f8138fb0 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f809db40 con 0x7f10f8138bd0 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f809de20 con 0x7f10f8139420 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10ff7b4640 10 monclient(hunting): authenticate will time out at 815.928793s 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.299+0000 7f10fcd4f640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f10f8138fb0 legacy=0x7f10f813df50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.300+0000 7f10fd550640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f10f8138bd0 0x7f10f80a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:41786/0 (socket says 172.21.15.121:41786) 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.300+0000 7f10fd550640 1 -- 172.21.15.121:0/176650870 learned_addr learned my addr 172.21.15.121:0/176650870 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.300+0000 7f10fdd51640 1 --1- 172.21.15.121:0/176650870 >> v1:172.21.15.153:6791/0 conn(0x7f10f8139420 0x7f10f8141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:46734/0 (socket says 172.21.15.121:46734) 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10fdd51640 1 -- 172.21.15.121:0/176650870 >> v1:172.21.15.153:6791/0 conn(0x7f10f8139420 legacy=0x7f10f8141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10fd550640 1 -- 172.21.15.121:0/176650870 >> v1:172.21.15.153:6790/0 conn(0x7f10f8138bd0 legacy=0x7f10f80a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645358800 0 0) 0x7f10f809de20 con 0x7f10f8139420 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient(hunting): my global_id is 4496 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e4003520 con 0x7f10f8139420 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1778321501 0 0) 0x7f10e4003520 con 0x7f10f8139420 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 >> v1:172.21.15.153:6790/0 conn(0x7f10f8138bd0 legacy=0x7f10f80a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 >> v1:172.21.15.153:6789/0 conn(0x7f10f8138fb0 legacy=0x7f10f813df50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:16.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 monclient: found mon.noname-h 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.153:6791/0 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f80a5da0 con 0x7f10f8139420 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f10f0001f20 con 0x7f10f8139420 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10ff7b4640 5 monclient: authenticate success, global_id 4496 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10ff7b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:52:16.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: dump: 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.309 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10ff7b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 <== mon.5 v1:172.21.15.153:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10f0004b60 con 0x7f10f8139420 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 1 -- 172.21.15.121:0/176650870 <== mon.5 v1:172.21.15.153:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f10f0005110 con 0x7f10f8139420 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10ff7b4640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10ff7b4640 4 set_mon_vals no callback set 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.302+0000 7f10effff640 10 monclient: dump: 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.310 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.311 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 10 monclient: shutdown 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 1 -- 172.21.15.121:0/176650870 >> v1:172.21.15.153:6791/0 conn(0x7f10f8139420 legacy=0x7f10f8141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 1 -- 172.21.15.121:0/176650870 shutdown_connections 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 1 -- 172.21.15.121:0/176650870 >> 172.21.15.121:0/176650870 conn(0x7f10f80ab800 msgr2=0x7f10f80abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 1 -- 172.21.15.121:0/176650870 shutdown_connections 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.303+0000 7f10ff7b4640 1 -- 172.21.15.121:0/176650870 wait complete. 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient: monmap: 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:16.312 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 1 Processor -- start 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 1 -- start start 2024-09-13T13:52:16.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient: init 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient: _add_conns ranks=[1,5,4,6,0,2,8,7,3] 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient(hunting): picked mon.noname-f con 0x7f10f8138bd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient(hunting): picked mon.noname-h con 0x7f10f8138fb0 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient(hunting): picked mon.noname-c con 0x7f10f8139420 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f81326c0 con 0x7f10f8138bd0 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f80a74d0 con 0x7f10f8139420 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f80a5a30 con 0x7f10f8138fb0 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10ff7b4640 10 monclient(hunting): authenticate will time out at 815.933929s 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.304+0000 7f10fd550640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f10f8138bd0 legacy=0x7f10f812e940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.305+0000 7f10fdd51640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f10f8139420 0x7f10f812f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:48698/0 (socket says 172.21.15.121:48698) 2024-09-13T13:52:16.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.305+0000 7f10fdd51640 1 -- 172.21.15.121:0/3740776998 learned_addr learned my addr 172.21.15.121:0/3740776998 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.307+0000 7f10fdd51640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.121:6791/0 conn(0x7f10f8139420 legacy=0x7f10f812f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.307+0000 7f10fcd4f640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6791/0 conn(0x7f10f8138fb0 legacy=0x7f10f812f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.307+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 DispatchQueue::entry inject delay of 0.002000 2024-09-13T13:52:16.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200627677 0 0) 0x7f10f80a5a30 con 0x7f10f8138fb0 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): my global_id is 4505 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10dc003520 con 0x7f10f8138fb0 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3005401567 0 0) 0x7f10dc003520 con 0x7f10f8138fb0 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f80a5a30 con 0x7f10f8138fb0 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f10e8004590 con 0x7f10f8138fb0 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): dump: 2024-09-13T13:52:16.318 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.319 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4115615809 0 0) 0x7f10f80a5a30 con 0x7f10f8138fb0 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.121:6791/0 conn(0x7f10f8139420 legacy=0x7f10f812f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6789/0 conn(0x7f10f8138bd0 legacy=0x7f10f812e940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 monclient: found mon.h 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f81323e0 con 0x7f10f8138fb0 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10ff7b4640 5 monclient: authenticate success, global_id 4505 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10ff7b4640 10 monclient: _renew_subs 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.309+0000 7f10ff7b4640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10f8131300 con 0x7f10f8138fb0 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10e8004870 con 0x7f10f8138fb0 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10edffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f10e8004e20 con 0x7f10f8138fb0 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10edffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10edffb640 10 monclient: dump: 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.321 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10ff7b4640 10 monclient: _renew_subs 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10ef7fe640 4 set_mon_vals no callback set 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10ff7b4640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.310+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f10f81d6fe0 con 0x7f10f8138fb0 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.311+0000 7f10ff7b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.311+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f10f8069380 con 0x7f10f8133550 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.311+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f10e8004fc0 con 0x7f10f8138fb0 2024-09-13T13:52:16.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.311+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.5 v1:172.21.15.153:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f10e804e230 con 0x7f10f8138fb0 2024-09-13T13:52:16.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.311+0000 7f10fdd51640 10 monclient: get_auth_request con 0x7f10dc03f970 auth_method 0 2024-09-13T13:52:16.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.313+0000 7f10fd550640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6790/0 conn(0x7f10f8133550 legacy=0x7f10f8133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.382+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 671621700 0 0) 0x7f10f8069380 con 0x7f10f8133550 2024-09-13T13:52:16.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.382+0000 7f10edffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.382+0000 7f10edffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.382+0000 7f10edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.382+0000 7f10edffb640 10 monclient(hunting): my global_id is 4458 2024-09-13T13:52:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.382+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f80a5a30 con 0x7f10f8133550 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1459557027 0 0) 0x7f10f80a5a30 con 0x7f10f8133550 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f8069380 con 0x7f10f8133550 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f10f4001aa0 con 0x7f10f8133550 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 10 monclient: dump: 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.436 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3140361539 0 0) 0x7f10f8069380 con 0x7f10f8133550 2024-09-13T13:52:16.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.434+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10f81310a0 con 0x7f10f8133550 2024-09-13T13:52:16.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.435+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f10f40058d0 con 0x7f10f8133550 2024-09-13T13:52:16.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.435+0000 7f10edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.435+0000 7f10edffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:16.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.435+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6790/0 conn(0x7f10f8133550 legacy=0x7f10f8133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.435+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6790/0 conn(0x7f10f8133550 legacy=0x7f10f8133930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.441+0000 7f10ff7b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:16.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.441+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f10f8069950 con 0x7f10f8134040 2024-09-13T13:52:16.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10fd550640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6790/0 conn(0x7f10f8134040 legacy=0x7f10f81db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593255499 0 0) 0x7f10f8069950 con 0x7f10f8134040 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient(hunting): my global_id is 4467 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f8069380 con 0x7f10f8134040 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2974525666 0 0) 0x7f10f8069380 con 0x7f10f8134040 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10f8069950 con 0x7f10f8134040 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f10f4004ee0 con 0x7f10f8134040 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 10 monclient: dump: 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.446 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.447 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 70858940 0 0) 0x7f10f8069950 con 0x7f10f8134040 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.444+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 --> v1:172.21.15.153:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10f8134420 con 0x7f10f8134040 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 <== mon.3 v1:172.21.15.153:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7536 (unknown 0 0 3949327537) 0x7f10f4005a80 con 0x7f10f8134040 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10edffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6790/0 conn(0x7f10f8134040 legacy=0x7f10f81db1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10edffb640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6790/0 conn(0x7f10f8134040 legacy=0x7f10f81db1d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:16.448 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 60068, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.449 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.450 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.451 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.453 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.454 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.455 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.456 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.457 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:16.458 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.459 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.460 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.461 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.462 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.463 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:16.464 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.465 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.466 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:16.467 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:16.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6812/1206688214 conn(0x7f10dc03f970 legacy=0x7f10dc041e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 10 monclient: shutdown 2024-09-13T13:52:16.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 >> v1:172.21.15.153:6791/0 conn(0x7f10f8138fb0 legacy=0x7f10f812f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10fdd51640 1 -- 172.21.15.121:0/3740776998 reap_dead start 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10fdd51640 1 -- 172.21.15.121:0/3740776998 reap_dead start 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 shutdown_connections 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 >> 172.21.15.121:0/3740776998 conn(0x7f10f80ab800 msgr2=0x7f10f812c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 shutdown_connections 2024-09-13T13:52:16.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.445+0000 7f10ff7b4640 1 -- 172.21.15.121:0/3740776998 wait complete. 2024-09-13T13:52:16.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:16.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-09-13T13:52:16.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-13T13:52:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.567+0000 7ffb59924640 10 monclient: get_monmap_and_config 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.567+0000 7ffb59924640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.567+0000 7ffb59924640 10 monclient: monmap: 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:16.568596+0000 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:16.568596+0000 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.570 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.568+0000 7ffb59924640 1 Processor -- start 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.568+0000 7ffb59924640 1 -- start start 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.568+0000 7ffb59924640 10 monclient: init 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.568+0000 7ffb59924640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.568+0000 7ffb59924640 10 monclient: _add_conns ranks=[7,2,8,6,0,4,3,5,1] 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 10 monclient(hunting): picked mon.noname-i con 0x7ffb540a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 10 monclient(hunting): picked mon.noname-b con 0x7ffb5409d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 10 monclient(hunting): picked mon.noname-e con 0x7ffb540a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb5409e210 con 0x7ffb5409d150 2024-09-13T13:52:16.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb540a7ef0 con 0x7ffb540a6160 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb5409d970 con 0x7ffb540a6c50 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb59924640 10 monclient(hunting): authenticate will time out at 816.198069s 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb537fe640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7ffb540a6c50 0x7ffb54141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:50594/0 (socket says 172.21.15.121:50594) 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb527fc640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7ffb5409d150 0x7ffb5413e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:43426/0 (socket says 172.21.15.121:43426) 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb537fe640 1 -- 172.21.15.121:0/278650937 learned_addr learned my addr 172.21.15.121:0/278650937 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.569+0000 7ffb52ffd640 1 -- 172.21.15.121:0/278650937 >> v1:172.21.15.153:6792/0 conn(0x7ffb540a6160 legacy=0x7ffb540a6540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6792/0 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb537fe640 1 -- 172.21.15.121:0/278650937 >> v1:172.21.15.121:6793/0 conn(0x7ffb540a6c50 legacy=0x7ffb54141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb527fc640 1 -- 172.21.15.121:0/278650937 >> v1:172.21.15.121:6790/0 conn(0x7ffb5409d150 legacy=0x7ffb5413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878033868 0 0) 0x7ffb5409e210 con 0x7ffb5409d150 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient(hunting): my global_id is 4502 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb3c003520 con 0x7ffb5409d150 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 383831449 0 0) 0x7ffb3c003520 con 0x7ffb5409d150 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 >> v1:172.21.15.121:6793/0 conn(0x7ffb540a6c50 legacy=0x7ffb54141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 >> v1:172.21.15.153:6792/0 conn(0x7ffb540a6160 legacy=0x7ffb540a6540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 monclient: found mon.noname-b 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5409e630 con 0x7ffb5409d150 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb59924640 5 monclient: authenticate success, global_id 4502 2024-09-13T13:52:16.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb48001ae0 con 0x7ffb5409d150 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb59924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: dump: 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.577 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb59924640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb480047e0 con 0x7ffb5409d150 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb59924640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 1 -- 172.21.15.121:0/278650937 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb48004d50 con 0x7ffb5409d150 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb59924640 4 set_mon_vals no callback set 2024-09-13T13:52:16.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.571+0000 7ffb51ffb640 10 monclient: dump: 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.579 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 10 monclient: shutdown 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 1 -- 172.21.15.121:0/278650937 >> v1:172.21.15.121:6790/0 conn(0x7ffb5409d150 legacy=0x7ffb5413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 1 -- 172.21.15.121:0/278650937 shutdown_connections 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 1 -- 172.21.15.121:0/278650937 >> 172.21.15.121:0/278650937 conn(0x7ffb540ab940 msgr2=0x7ffb540abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 1 -- 172.21.15.121:0/278650937 shutdown_connections 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 1 -- 172.21.15.121:0/278650937 wait complete. 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 10 monclient: build_initial_monmap 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.572+0000 7ffb59924640 10 monclient: monmap: 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:16.580 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 1 Processor -- start 2024-09-13T13:52:16.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 1 -- start start 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient: init 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient: _add_conns ranks=[6,0,3,4,2,5,7,8,1] 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient(hunting): picked mon.noname-d con 0x7ffb5409d150 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient(hunting): picked mon.noname-a con 0x7ffb540a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient(hunting): picked mon.noname-g con 0x7ffb540a6c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb540a4560 con 0x7ffb540a6160 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb541246d0 con 0x7ffb540a6c50 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb54126190 con 0x7ffb5409d150 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb59924640 10 monclient(hunting): authenticate will time out at 816.202903s 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb527fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ffb540a6160 0x7ffb54139070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:38428/0 (socket says 172.21.15.121:38428) 2024-09-13T13:52:16.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.573+0000 7ffb527fc640 1 -- 172.21.15.121:0/383179268 learned_addr learned my addr 172.21.15.121:0/383179268 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb52ffd640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.121:6792/0 conn(0x7ffb5409d150 legacy=0x7ffb541cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb527fc640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.121:6789/0 conn(0x7ffb540a6160 legacy=0x7ffb54139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635134338 0 0) 0x7ffb54126190 con 0x7ffb5409d150 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): my global_id is 4569 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb40003520 con 0x7ffb5409d150 2024-09-13T13:52:16.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3067713913 0 0) 0x7ffb540a4560 con 0x7ffb540a6160 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): my global_id is 4590 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb54126190 con 0x7ffb540a6160 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb537fe640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6790/0 conn(0x7ffb540a6c50 legacy=0x7ffb541d59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2281246564 0 0) 0x7ffb40003520 con 0x7ffb5409d150 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb52ffd640 1 --1- 172.21.15.121:0/383179268 >> v1:172.21.15.121:6792/0 conn(0x7ffb5409d150 0x7ffb541cd680 :-1 s=OPENED pgs=117 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7ffb44004650 mon_map magic: 0 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb540a4560 con 0x7ffb5409d150 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 557443155 0 0) 0x7ffb54126190 con 0x7ffb540a6160 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb40003520 con 0x7ffb540a6160 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb48004cd0 con 0x7ffb540a6160 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:16.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): dump: 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.588 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb44004650 con 0x7ffb5409d150 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient(hunting): dump: 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.589 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2731268673 0 0) 0x7ffb40003520 con 0x7ffb540a6160 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.121:6792/0 conn(0x7ffb5409d150 legacy=0x7ffb541cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6790/0 conn(0x7ffb540a6c50 legacy=0x7ffb541d59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: _finish_hunting 0 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 monclient: found mon.a 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb54139780 con 0x7ffb540a6160 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: _finish_auth 0 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: _check_auth_tickets 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 5 monclient: authenticate success, global_id 4590 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2255110328 0 0) 0x7ffb540a4560 con 0x7ffb5409d150 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 10 monclient: _renew_subs 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:16.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffb541399d0 con 0x7ffb540a6160 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 10 monclient: _renew_subs 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ffb54123dd0 con 0x7ffb540a6160 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb480047e0 con 0x7ffb540a6160 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb48001a80 con 0x7ffb540a6160 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.576+0000 7ffb517fa640 4 set_mon_vals no callback set 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.577+0000 7ffb3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.577+0000 7ffb3bfff640 10 monclient: dump: 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.592 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.577+0000 7ffb59924640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.577+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffb340065c0 con 0x7ffb34002860 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.577+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7ffb48011b60 con 0x7ffb540a6160 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.577+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7ffb4804e1b0 con 0x7ffb540a6160 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.578+0000 7ffb537fe640 10 monclient: get_auth_request con 0x7ffb40040030 auth_method 0 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb52ffd640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6791/0 conn(0x7ffb34002860 legacy=0x7ffb34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047477711 0 0) 0x7ffb340065c0 con 0x7ffb34002860 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient(hunting): my global_id is 4514 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb540a4560 con 0x7ffb34002860 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 639127382 0 0) 0x7ffb540a4560 con 0x7ffb34002860 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb340065c0 con 0x7ffb34002860 2024-09-13T13:52:16.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb44004e30 con 0x7ffb34002860 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 10 monclient: dump: 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.595 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3860703281 0 0) 0x7ffb340065c0 con 0x7ffb34002860 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.580+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffb3400a160 con 0x7ffb34002860 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.581+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ffb44005140 con 0x7ffb34002860 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.581+0000 7ffb3bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.581+0000 7ffb3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.581+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6791/0 conn(0x7ffb34002860 legacy=0x7ffb34004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.581+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6791/0 conn(0x7ffb34002860 legacy=0x7ffb34004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.587+0000 7ffb59924640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.587+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ffb34009e50 con 0x7ffb34009110 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb52ffd640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6791/0 conn(0x7ffb34009110 legacy=0x7ffb340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2124904479 0 0) 0x7ffb34009e50 con 0x7ffb34009110 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb3bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb3bfff640 10 monclient(hunting): my global_id is 4523 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.589+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb340065c0 con 0x7ffb34009110 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3134323244 0 0) 0x7ffb340065c0 con 0x7ffb34009110 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb34009e50 con 0x7ffb34009110 2024-09-13T13:52:16.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ffb44006000 con 0x7ffb34009110 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 10 monclient: dump: 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:16.599 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2782526130 0 0) 0x7ffb34009e50 con 0x7ffb34009110 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 --> v1:172.21.15.153:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffb3400c3f0 con 0x7ffb34009110 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 <== mon.5 v1:172.21.15.153:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7872 (unknown 0 0 2765274512) 0x7ffb440081e0 con 0x7ffb34009110 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6791/0 conn(0x7ffb34009110 legacy=0x7ffb340095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.590+0000 7ffb3bfff640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6791/0 conn(0x7ffb34009110 legacy=0x7ffb340095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 22, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 60222, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.602 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.603 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:16.604 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:16.605 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.607 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.608 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:16.610 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.611 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.612 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:16.613 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.615 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:16.616 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:16.617 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.618 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.619 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:16.620 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:16.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.153:6812/1206688214 conn(0x7ffb40040030 legacy=0x7ffb400424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 10 monclient: shutdown 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 >> v1:172.21.15.121:6789/0 conn(0x7ffb540a6160 legacy=0x7ffb54139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb537fe640 1 -- 172.21.15.121:0/383179268 reap_dead start 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb537fe640 1 -- 172.21.15.121:0/383179268 reap_dead start 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 shutdown_connections 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 >> 172.21.15.121:0/383179268 conn(0x7ffb540ab940 msgr2=0x7ffb541c11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 shutdown_connections 2024-09-13T13:52:16.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:16.591+0000 7ffb59924640 1 -- 172.21.15.121:0/383179268 wait complete. 2024-09-13T13:52:16.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-13T13:52:16.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-13T13:52:36.623 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-09-13T13:52:36.624 INFO:tasks.ceph.mon.f:Restarting daemon 2024-09-13T13:52:36.624 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-09-13T13:52:36.627 INFO:tasks.ceph.mon.f:Started 2024-09-13T13:52:36.627 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-09-13T13:52:36.627 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-13T13:52:36.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-13T13:52:36.630 INFO:tasks.ceph.mon.i:Started 2024-09-13T13:52:36.630 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-09-13T13:52:36.630 INFO:tasks.ceph.mon.g:Sent signal 19 2024-09-13T13:52:36.630 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs to unfreeze mons 2024-09-13T13:52:56.631 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-09-13T13:52:56.632 INFO:tasks.ceph.mon.g:Sent signal 18 2024-09-13T13:52:56.632 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-13T13:52:56.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T13:52:56.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.742+0000 7f106c0bf640 10 monclient: get_monmap_and_config 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.742+0000 7f106c0bf640 10 monclient: build_initial_monmap 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.742+0000 7f106c0bf640 10 monclient: monmap: 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:56.744525+0000 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:56.744525+0000 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:56.745 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:56.746 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.743+0000 7f106c0bf640 1 Processor -- start 2024-09-13T13:52:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.743+0000 7f106c0bf640 1 -- start start 2024-09-13T13:52:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.743+0000 7f106c0bf640 10 monclient: init 2024-09-13T13:52:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient: _add_conns ranks=[6,5,2,8,0,4,7,1,3] 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient(hunting): picked mon.noname-d con 0x7f106409c950 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient(hunting): picked mon.noname-h con 0x7f1064135b50 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient(hunting): picked mon.noname-b con 0x7f106413f660 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1064135f50 con 0x7f106413f660 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106409d860 con 0x7f1064135b50 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106409db40 con 0x7f106409c950 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106c0bf640 10 monclient(hunting): authenticate will time out at 856.374236s 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106a65c640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f106413f660 0x7f1064141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:54968/0 (socket says 172.21.15.121:54968) 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f1069e5b640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f106409c950 0x7f10640a7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51384/0 (socket says 172.21.15.121:51384) 2024-09-13T13:52:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.744+0000 7f106a65c640 1 -- 172.21.15.121:0/416217977 learned_addr learned my addr 172.21.15.121:0/416217977 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f106a65c640 1 -- 172.21.15.121:0/416217977 >> v1:172.21.15.121:6790/0 conn(0x7f106413f660 legacy=0x7f1064141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1069e5b640 1 -- 172.21.15.121:0/416217977 >> v1:172.21.15.121:6792/0 conn(0x7f106409c950 legacy=0x7f10640a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f106965a640 1 -- 172.21.15.121:0/416217977 >> v1:172.21.15.153:6791/0 conn(0x7f1064135b50 legacy=0x7f106413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617480421 0 0) 0x7f106409db40 con 0x7f106409c950 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): _negotiate 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:56.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): my global_id is 4578 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1050003520 con 0x7f106409c950 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700682028 0 0) 0x7f1064135f50 con 0x7f106413f660 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): _negotiate 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 10 monclient(hunting): my global_id is 4511 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f106409db40 con 0x7f106413f660 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1450747326 0 0) 0x7f1050003520 con 0x7f106409c950 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.746+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 >> v1:172.21.15.153:6791/0 conn(0x7f1064135b50 legacy=0x7f106413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 >> v1:172.21.15.121:6790/0 conn(0x7f106413f660 legacy=0x7f1064141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: _finish_hunting 0 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 1 monclient: found mon.noname-d 2024-09-13T13:52:56.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106409de20 con 0x7f106409c950 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: _finish_auth 0 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: _check_auth_tickets 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 5 monclient: authenticate success, global_id 4578 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1060001d90 con 0x7f106409c950 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: dump: 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:56.754 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:56.755 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1060002060 con 0x7f106409c950 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 1 -- 172.21.15.121:0/416217977 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1060004ea0 con 0x7f106409c950 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 4 set_mon_vals no callback set 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f1068e59640 10 monclient: dump: 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:56.756 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:56.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 10 monclient: shutdown 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 1 -- 172.21.15.121:0/416217977 >> v1:172.21.15.121:6792/0 conn(0x7f106409c950 legacy=0x7f10640a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 1 -- 172.21.15.121:0/416217977 shutdown_connections 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 1 -- 172.21.15.121:0/416217977 >> 172.21.15.121:0/416217977 conn(0x7f10640ab760 msgr2=0x7f10640abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 1 -- 172.21.15.121:0/416217977 shutdown_connections 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.747+0000 7f106c0bf640 1 -- 172.21.15.121:0/416217977 wait complete. 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.748+0000 7f106c0bf640 10 monclient: build_initial_monmap 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.748+0000 7f106c0bf640 10 monclient: monmap: 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:56.759 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.748+0000 7f106c0bf640 1 Processor -- start 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.748+0000 7f106c0bf640 1 -- start start 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.748+0000 7f106c0bf640 10 monclient: init 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient: _add_conns ranks=[1,5,3,6,4,2,8,0,7] 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient(hunting): picked mon.noname-f con 0x7f106409c950 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient(hunting): picked mon.noname-h con 0x7f1064135b50 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient(hunting): picked mon.noname-g con 0x7f106413f660 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106413e180 con 0x7f106409c950 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106413e460 con 0x7f106413f660 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1064142d00 con 0x7f1064135b50 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106c0bf640 10 monclient(hunting): authenticate will time out at 856.379234s 2024-09-13T13:52:56.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106965a640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f1064135b50 0x7f106413ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:40998/0 (socket says 172.21.15.121:40998) 2024-09-13T13:52:56.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106a65c640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7f106413f660 0x7f10641d60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:51722/0 (socket says 172.21.15.121:51722) 2024-09-13T13:52:56.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.749+0000 7f106965a640 1 -- 172.21.15.121:0/2224740280 learned_addr learned my addr 172.21.15.121:0/2224740280 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:56.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.751+0000 7f1069e5b640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6789/0 conn(0x7f106409c950 legacy=0x7f106413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:56.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.751+0000 7f106a65c640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6790/0 conn(0x7f106413f660 legacy=0x7f10641d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:56.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:56.751+0000 7f106965a640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6791/0 conn(0x7f1064135b50 legacy=0x7f106413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641176523 0 0) 0x7f1064142d00 con 0x7f1064135b50 2024-09-13T13:52:57.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): my global_id is 4532 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1048003520 con 0x7f1064135b50 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727246057 0 0) 0x7f106413e460 con 0x7f106413f660 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 10 monclient(hunting): my global_id is 4485 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.382+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1064142d00 con 0x7f106413f660 2024-09-13T13:52:57.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3328713708 0 0) 0x7f1048003520 con 0x7f1064135b50 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f106413e460 con 0x7f1064135b50 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1054003610 con 0x7f1064135b50 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.389 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 218832493 0 0) 0x7f1064142d00 con 0x7f106413f660 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1048003520 con 0x7f106413f660 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f105c001c90 con 0x7f106413f660 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:57.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.391 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740076799 0 0) 0x7f106413e180 con 0x7f106409c950 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient(hunting): my global_id is 104104 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1064142d00 con 0x7f106409c950 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4256554167 0 0) 0x7f106413e460 con 0x7f1064135b50 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6790/0 conn(0x7f106413f660 legacy=0x7f10641d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6789/0 conn(0x7f106409c950 legacy=0x7f106413a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:52:57.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 monclient: found mon.h 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10640a9cd0 con 0x7f1064135b50 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: _finish_auth 0 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 5 monclient: authenticate success, global_id 4532 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 10 monclient: _renew_subs 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f106413b470 con 0x7f1064135b50 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 10 monclient: _renew_subs 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:57.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f106413cf30 con 0x7f1064135b50 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1054003b10 con 0x7f1064135b50 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1054004040 con 0x7f1064135b50 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105a7fc640 10 monclient: dump: 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.394 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.383+0000 7f105bfff640 4 set_mon_vals no callback set 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.384+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f10540041e0 con 0x7f1064135b50 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.384+0000 7f106c0bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.384+0000 7f106c0bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.384+0000 7f106c0bf640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:57.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.384+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104c005080 con 0x7f1064135b50 2024-09-13T13:52:57.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.384+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f105404e1c0 con 0x7f1064135b50 2024-09-13T13:52:57.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.385+0000 7f1069e5b640 10 monclient: get_auth_request con 0x7f1048040370 auth_method 0 2024-09-13T13:52:57.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.389+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f1054017890 con 0x7f1064135b50 2024-09-13T13:52:57.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.390+0000 7f105a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.390+0000 7f105a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:57.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.638+0000 7f106c0bf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.638+0000 7f106c0bf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.638+0000 7f106c0bf640 10 monclient: _send_mon_message to mon.h at v1:172.21.15.153:6791/0 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.638+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 --> v1:172.21.15.153:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f104c005370 con 0x7f1064135b50 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.639+0000 7f105a7fc640 1 -- 172.21.15.121:0/2224740280 <== mon.5 v1:172.21.15.153:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2744 (unknown 254746657 0 690991855) 0x7f1054017190 con 0x7f1064135b50 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.639+0000 7f105a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.639+0000 7f105a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:57.641 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":28,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"d","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","modified":"2024-09-13T13:50:47.391384Z","created":"2024-09-13T13:50:47.391384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6791","nonce":0}]},"addr":"172.21.15.121:6791/0","public_addr":"172.21.15.121:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6791","nonce":0}]},"addr":"172.21.15.153:6791/0","public_addr":"172.21.15.153:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6792","nonce":0}]},"addr":"172.21.15.121:6792/0","public_addr":"172.21.15.121:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6792","nonce":0}]},"addr":"172.21.15.153:6792/0","public_addr":"172.21.15.153:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6793","nonce":0}]},"addr":"172.21.15.121:6793/0","public_addr":"172.21.15.121:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T13:52:57.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1048040370 legacy=0x7f1048042830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 10 monclient: shutdown 2024-09-13T13:52:57.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:57.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 >> v1:172.21.15.153:6791/0 conn(0x7f1064135b50 legacy=0x7f106413ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 shutdown_connections 2024-09-13T13:52:57.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 >> 172.21.15.121:0/2224740280 conn(0x7f10640ab760 msgr2=0x7f10640a6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:57.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 shutdown_connections 2024-09-13T13:52:57.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.641+0000 7f106c0bf640 1 -- 172.21.15.121:0/2224740280 wait complete. 2024-09-13T13:52:57.654 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-13T13:52:57.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-13T13:52:57.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.751+0000 7fdeaa7cd640 10 monclient: get_monmap_and_config 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.751+0000 7fdeaa7cd640 10 monclient: build_initial_monmap 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.751+0000 7fdeaa7cd640 10 monclient: monmap: 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:57.752982+0000 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:57.752982+0000 2024-09-13T13:52:57.753 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:57.754 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 1 Processor -- start 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 1 -- start start 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient: init 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient: _add_conns ranks=[0,8,5,2,3,6,7,4,1] 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient(hunting): picked mon.noname-a con 0x7fdea40a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient(hunting): picked mon.noname-e con 0x7fdea409d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient(hunting): picked mon.noname-h con 0x7fdea40a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea409e210 con 0x7fdea40a6160 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea40a7ef0 con 0x7fdea40a6c50 2024-09-13T13:52:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea409d970 con 0x7fdea409d150 2024-09-13T13:52:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdeaa7cd640 10 monclient(hunting): authenticate will time out at 857.382604s 2024-09-13T13:52:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdea3fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdea40a6160 0x7fdea40a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59020/0 (socket says 172.21.15.121:59020) 2024-09-13T13:52:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.752+0000 7fdea3fff640 1 -- 172.21.15.121:0/3898029132 learned_addr learned my addr 172.21.15.121:0/3898029132 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea37fe640 1 -- 172.21.15.121:0/3898029132 >> v1:172.21.15.121:6793/0 conn(0x7fdea409d150 legacy=0x7fdea413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea3fff640 1 -- 172.21.15.121:0/3898029132 >> v1:172.21.15.121:6789/0 conn(0x7fdea40a6160 legacy=0x7fdea40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476905987 0 0) 0x7fdea409d970 con 0x7fdea409d150 2024-09-13T13:52:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.754+0000 7fdea2ffd640 10 monclient(hunting): my global_id is 4580 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde8c003520 con 0x7fdea409d150 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069617055 0 0) 0x7fdea409e210 con 0x7fdea40a6160 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient(hunting): my global_id is 4599 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea409d970 con 0x7fdea40a6160 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea8d6a640 1 -- 172.21.15.121:0/3898029132 >> v1:172.21.15.153:6791/0 conn(0x7fdea40a6c50 legacy=0x7fdea4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2765904384 0 0) 0x7fde8c003520 con 0x7fdea409d150 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 >> v1:172.21.15.153:6791/0 conn(0x7fdea40a6c50 legacy=0x7fdea4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 >> v1:172.21.15.121:6789/0 conn(0x7fdea40a6160 legacy=0x7fdea40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 monclient: found mon.noname-e 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea409e630 con 0x7fdea409d150 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: _finish_auth 0 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde98001b30 con 0x7fdea409d150 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: dump: 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdeaa7cd640 5 monclient: authenticate success, global_id 4580 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdeaa7cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde98003610 con 0x7fdea409d150 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdeaa7cd640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 1 -- 172.21.15.121:0/3898029132 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde98003b80 con 0x7fdea409d150 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdeaa7cd640 4 set_mon_vals no callback set 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.755+0000 7fdea2ffd640 10 monclient: dump: 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 10 monclient: shutdown 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/3898029132 >> v1:172.21.15.121:6793/0 conn(0x7fdea409d150 legacy=0x7fdea413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/3898029132 shutdown_connections 2024-09-13T13:52:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/3898029132 >> 172.21.15.121:0/3898029132 conn(0x7fdea40ab940 msgr2=0x7fdea40abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/3898029132 shutdown_connections 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/3898029132 wait complete. 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 10 monclient: build_initial_monmap 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 10 monclient: monmap: 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.756+0000 7fdeaa7cd640 1 Processor -- start 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 1 -- start start 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient: init 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient: _add_conns ranks=[6,4,2,8,0,7,5,1,3] 2024-09-13T13:52:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient(hunting): picked mon.noname-d con 0x7fdea409d150 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient(hunting): picked mon.noname-c con 0x7fdea40a6160 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient(hunting): picked mon.noname-b con 0x7fdea40a6c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea40a46a0 con 0x7fdea40a6c50 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea41247f0 con 0x7fdea40a6160 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea4126b50 con 0x7fdea409d150 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdeaa7cd640 10 monclient(hunting): authenticate will time out at 857.387705s 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdea3fff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fdea409d150 0x7fdea4139190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51392/0 (socket says 172.21.15.121:51392) 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.757+0000 7fdea3fff640 1 -- 172.21.15.121:0/2463928906 learned_addr learned my addr 172.21.15.121:0/2463928906 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.759+0000 7fdea8d6a640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6790/0 conn(0x7fdea40a6c50 legacy=0x7fdea41d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.759+0000 7fdea37fe640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6791/0 conn(0x7fdea40a6160 legacy=0x7fdea41d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.759+0000 7fdea3fff640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6792/0 conn(0x7fdea409d150 legacy=0x7fdea4139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3226741992 0 0) 0x7fdea41247f0 con 0x7fdea40a6160 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): my global_id is 4486 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde90003520 con 0x7fdea40a6160 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4122206827 0 0) 0x7fdea40a46a0 con 0x7fdea40a6c50 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): my global_id is 4520 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea41247f0 con 0x7fdea40a6c50 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 223426475 0 0) 0x7fdea4126b50 con 0x7fdea409d150 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): my global_id is 4587 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea40a46a0 con 0x7fdea409d150 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3663204669 0 0) 0x7fde90003520 con 0x7fdea40a6160 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea4126b50 con 0x7fdea40a6160 2024-09-13T13:52:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde98003b00 con 0x7fdea40a6160 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): dump: 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4235333500 0 0) 0x7fdea41247f0 con 0x7fdea40a6c50 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde90003520 con 0x7fdea40a6c50 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2058570308 0 0) 0x7fdea40a46a0 con 0x7fdea409d150 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea41247f0 con 0x7fdea409d150 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde9c002030 con 0x7fdea40a6c50 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): dump: 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde940047e0 con 0x7fdea409d150 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient(hunting): dump: 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 958148064 0 0) 0x7fdea4126b50 con 0x7fdea40a6160 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6792/0 conn(0x7fdea409d150 legacy=0x7fdea4139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6790/0 conn(0x7fdea40a6c50 legacy=0x7fdea41d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 monclient: found mon.c 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea41262b0 con 0x7fdea40a6160 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: _finish_auth 0 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 5 monclient: authenticate success, global_id 4486 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 10 monclient: _renew_subs 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdea41398a0 con 0x7fdea40a6160 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 10 monclient: _renew_subs 2024-09-13T13:52:57.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fdea41cd9d0 con 0x7fdea40a6160 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fde98001370 con 0x7fdea40a6160 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde98003610 con 0x7fdea40a6160 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea27fc640 4 set_mon_vals no callback set 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.760+0000 7fdea0ff9640 10 monclient: dump: 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.761+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fde98011ca0 con 0x7fdea40a6160 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.761+0000 7fdeaa7cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.761+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fde840065c0 con 0x7fde84002860 2024-09-13T13:52:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.761+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fde9804e120 con 0x7fdea40a6160 2024-09-13T13:52:57.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.762+0000 7fdea8d6a640 10 monclient: get_auth_request con 0x7fde90040710 auth_method 0 2024-09-13T13:52:57.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.763+0000 7fdea3fff640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6789/0 conn(0x7fde84002860 legacy=0x7fde84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2743242450 0 0) 0x7fde840065c0 con 0x7fde84002860 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient(hunting): my global_id is 4608 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea4126b50 con 0x7fde84002860 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2086664568 0 0) 0x7fdea4126b50 con 0x7fde84002860 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde840065c0 con 0x7fde84002860 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde94004eb0 con 0x7fde84002860 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient: dump: 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1416747400 0 0) 0x7fde840065c0 con 0x7fde84002860 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde8400a160 con 0x7fde84002860 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fde940063f0 con 0x7fde84002860 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6789/0 conn(0x7fde84002860 legacy=0x7fde84004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.764+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6789/0 conn(0x7fde84002860 legacy=0x7fde84004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.770+0000 7fdeaa7cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.770+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fde84009e70 con 0x7fde84009130 2024-09-13T13:52:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea3fff640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6789/0 conn(0x7fde84009130 legacy=0x7fde840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1023080433 0 0) 0x7fde84009e70 con 0x7fde84009130 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient(hunting): my global_id is 4617 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde840065c0 con 0x7fde84009130 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 40087713 0 0) 0x7fde840065c0 con 0x7fde84009130 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fde84009e70 con 0x7fde84009130 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fde94005270 con 0x7fde84009130 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient: dump: 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4114914698 0 0) 0x7fde84009e70 con 0x7fde84009130 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 --> v1:172.21.15.121:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde8400c410 con 0x7fde84009130 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 <== mon.0 v1:172.21.15.121:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 4045958596) 0x7fde94005720 con 0x7fde84009130 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6789/0 conn(0x7fde84009130 legacy=0x7fde840095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.773+0000 7fdea0ff9640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6789/0 conn(0x7fde84009130 legacy=0x7fde840095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:57.793 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 101431, 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:57.794 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:57.795 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:57.796 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:57.798 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:57.799 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.800 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.801 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.802 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:57.803 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.804 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.805 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:57.807 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.808 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.809 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.811 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.812 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.813 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.814 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:57.815 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:57.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.153:6812/1206688214 conn(0x7fde90040710 legacy=0x7fde90042bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 10 monclient: shutdown 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 >> v1:172.21.15.121:6791/0 conn(0x7fdea40a6160 legacy=0x7fdea41d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdea8d6a640 1 -- 172.21.15.121:0/2463928906 reap_dead start 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdea8d6a640 1 -- 172.21.15.121:0/2463928906 reap_dead start 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 shutdown_connections 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 >> 172.21.15.121:0/2463928906 conn(0x7fdea40ab940 msgr2=0x7fdea41c1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 shutdown_connections 2024-09-13T13:52:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.774+0000 7fdeaa7cd640 1 -- 172.21.15.121:0/2463928906 wait complete. 2024-09-13T13:52:57.817 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:57.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-13T13:52:57.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.914+0000 7f11a7e71640 10 monclient: get_monmap_and_config 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.914+0000 7f11a7e71640 10 monclient: build_initial_monmap 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.914+0000 7f11a7e71640 10 monclient: monmap: 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:57.915957+0000 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:57.915957+0000 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:57.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:57.917 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.914+0000 7f11a7e71640 1 Processor -- start 2024-09-13T13:52:57.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.914+0000 7f11a7e71640 1 -- start start 2024-09-13T13:52:57.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient: init 2024-09-13T13:52:57.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:57.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient: _add_conns ranks=[7,6,2,3,1,5,8,0,4] 2024-09-13T13:52:57.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient(hunting): picked mon.noname-i con 0x7f11a00a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient(hunting): picked mon.noname-d con 0x7f11a009d150 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient(hunting): picked mon.noname-b con 0x7f11a00a6c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a009e210 con 0x7f11a00a6c50 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a00a7ef0 con 0x7f11a009d150 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a009d970 con 0x7f11a00a6160 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a7e71640 10 monclient(hunting): authenticate will time out at 857.545542s 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a540c640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f11a009d150 0x7f11a013e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51398/0 (socket says 172.21.15.121:51398) 2024-09-13T13:52:57.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.915+0000 7f11a540c640 1 -- 172.21.15.121:0/4087780871 learned_addr learned my addr 172.21.15.121:0/4087780871 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a540c640 1 -- 172.21.15.121:0/4087780871 >> v1:172.21.15.121:6792/0 conn(0x7f11a009d150 legacy=0x7f11a013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a640e640 1 -- 172.21.15.121:0/4087780871 >> v1:172.21.15.121:6790/0 conn(0x7f11a00a6c50 legacy=0x7f11a0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871925104 0 0) 0x7f11a00a7ef0 con 0x7f11a009d150 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): my global_id is 4596 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f118c003520 con 0x7f11a009d150 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268988302 0 0) 0x7f11a009e210 con 0x7f11a00a6c50 2024-09-13T13:52:57.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.917+0000 7f11a4c0b640 10 monclient(hunting): my global_id is 4529 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11a00a7ef0 con 0x7f11a00a6c50 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a5c0d640 1 -- 172.21.15.121:0/4087780871 >> v1:172.21.15.153:6792/0 conn(0x7f11a00a6160 legacy=0x7f11a00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 63614725 0 0) 0x7f118c003520 con 0x7f11a009d150 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 >> v1:172.21.15.153:6792/0 conn(0x7f11a00a6160 legacy=0x7f11a00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 >> v1:172.21.15.121:6790/0 conn(0x7f11a00a6c50 legacy=0x7f11a0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: _finish_hunting 0 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 monclient: found mon.noname-d 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:52:57.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a009e630 con 0x7f11a009d150 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: _finish_auth 0 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: _check_auth_tickets 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1190001ae0 con 0x7f11a009d150 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 5 monclient: authenticate success, global_id 4596 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: dump: 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.924 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:57.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11900048a0 con 0x7f11a009d150 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 1 -- 172.21.15.121:0/4087780871 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1190004e50 con 0x7f11a009d150 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 4 set_mon_vals no callback set 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a4c0b640 10 monclient: dump: 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.926 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 10 monclient: shutdown 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 1 -- 172.21.15.121:0/4087780871 >> v1:172.21.15.121:6792/0 conn(0x7f11a009d150 legacy=0x7f11a013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 1 -- 172.21.15.121:0/4087780871 shutdown_connections 2024-09-13T13:52:57.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 1 -- 172.21.15.121:0/4087780871 >> 172.21.15.121:0/4087780871 conn(0x7f11a00ab940 msgr2=0x7f11a00abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 1 -- 172.21.15.121:0/4087780871 shutdown_connections 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.918+0000 7f11a7e71640 1 -- 172.21.15.121:0/4087780871 wait complete. 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.919+0000 7f11a7e71640 10 monclient: build_initial_monmap 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.919+0000 7f11a7e71640 10 monclient: monmap: 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:57.928 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:57.929 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:57.929 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:57.929 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:57.929 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:57.929 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.919+0000 7f11a7e71640 1 Processor -- start 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.919+0000 7f11a7e71640 1 -- start start 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.919+0000 7f11a7e71640 10 monclient: init 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient: _add_conns ranks=[4,0,7,3,5,2,8,1,6] 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient(hunting): picked mon.noname-c con 0x7f11a009d150 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient(hunting): picked mon.noname-a con 0x7f11a00a6160 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient(hunting): picked mon.noname-i con 0x7f11a00a6c50 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a00a89b0 con 0x7f11a00a6160 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a0124840 con 0x7f11a009d150 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a01268c0 con 0x7f11a00a6c50 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:57.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a7e71640 10 monclient(hunting): authenticate will time out at 857.550270s 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a5c0d640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f11a009d150 0x7f11a01391e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33528/0 (socket says 172.21.15.121:33528) 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.920+0000 7f11a5c0d640 1 -- 172.21.15.121:0/849918267 learned_addr learned my addr 172.21.15.121:0/849918267 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11a540c640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6789/0 conn(0x7f11a00a6160 legacy=0x7f11a01d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11a5c0d640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6791/0 conn(0x7f11a009d150 legacy=0x7f11a01391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11a640e640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.153:6792/0 conn(0x7f11a00a6c50 legacy=0x7f11a01d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2172637471 0 0) 0x7f11a009e630 con 0x7f11a00a6160 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): my global_id is 4626 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1184003550 con 0x7f11a00a6160 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038366549 0 0) 0x7f11a0124840 con 0x7f11a009d150 2024-09-13T13:52:57.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): my global_id is 4495 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11a009e630 con 0x7f11a009d150 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 712595458 0 0) 0x7f11a01268c0 con 0x7f11a00a6c50 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 10 monclient(hunting): my global_id is 104119 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11a0124840 con 0x7f11a00a6c50 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1970402430 0 0) 0x7f11a009e630 con 0x7f11a009d150 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11a01268c0 con 0x7f11a009d150 2024-09-13T13:52:57.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.922+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1621023932 0 0) 0x7f1184003550 con 0x7f11a00a6160 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11a009e630 con 0x7f11a00a6160 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f119c003610 con 0x7f11a009d150 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): dump: 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.937 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1190004d90 con 0x7f11a00a6160 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient(hunting): dump: 2024-09-13T13:52:57.938 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.939 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2859091414 0 0) 0x7f11a01268c0 con 0x7f11a009d150 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.153:6792/0 conn(0x7f11a00a6c50 legacy=0x7f11a01d6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6789/0 conn(0x7f11a00a6160 legacy=0x7f11a01d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: _finish_hunting 0 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 monclient: found mon.c 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a0126ba0 con 0x7f11a009d150 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: _finish_auth 0 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: _check_auth_tickets 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:57.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 5 monclient: authenticate success, global_id 4495 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 10 monclient: _renew_subs 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f11a0126300 con 0x7f11a009d150 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 10 monclient: _renew_subs 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f11a0125580 con 0x7f11a009d150 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f119c003b80 con 0x7f11a009d150 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f119c0040b0 con 0x7f11a009d150 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f1197fff640 4 set_mon_vals no callback set 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11967fc640 10 monclient: dump: 2024-09-13T13:52:57.941 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.942 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.923+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11880065c0 con 0x7f1188002860 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.924+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f119c004250 con 0x7f11a009d150 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.924+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f119c04e2e0 con 0x7f11a009d150 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.924+0000 7f11a640e640 10 monclient: get_auth_request con 0x7f1184040080 auth_method 0 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11a540c640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6790/0 conn(0x7f1188002860 legacy=0x7f1188004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152560090 0 0) 0x7f11880065c0 con 0x7f1188002860 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient(hunting): my global_id is 4538 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11a01268c0 con 0x7f1188002860 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 379677295 0 0) 0x7f11a01268c0 con 0x7f1188002860 2024-09-13T13:52:57.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11880065c0 con 0x7f1188002860 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f11900048a0 con 0x7f1188002860 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient: dump: 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.944 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 402745276 0 0) 0x7f11880065c0 con 0x7f1188002860 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f118800a160 con 0x7f1188002860 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1190005450 con 0x7f1188002860 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6790/0 conn(0x7f1188002860 legacy=0x7f1188004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.926+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6790/0 conn(0x7f1188002860 legacy=0x7f1188004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:57.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.933+0000 7f11a7e71640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:57.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.933+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1188009e50 con 0x7f1188009110 2024-09-13T13:52:57.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11a540c640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6790/0 conn(0x7f1188009110 legacy=0x7f11880095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:57.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2290453085 0 0) 0x7f1188009e50 con 0x7f1188009110 2024-09-13T13:52:57.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient(hunting): my global_id is 4547 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11880065c0 con 0x7f1188009110 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1171438869 0 0) 0x7f11880065c0 con 0x7f1188009110 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1188009e50 con 0x7f1188009110 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1190005b30 con 0x7f1188009110 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 10 monclient: dump: 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:57.948 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:57.949 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1888834257 0 0) 0x7f1188009e50 con 0x7f1188009110 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.935+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 --> v1:172.21.15.121:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f118800c3f0 con 0x7f1188009110 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 <== mon.2 v1:172.21.15.121:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 2334329192) 0x7f1190007040 con 0x7f1188009110 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11967fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6790/0 conn(0x7f1188009110 legacy=0x7f11880095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11967fc640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6790/0 conn(0x7f1188009110 legacy=0x7f11880095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 101603, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:57.970 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1184040080 legacy=0x7f1184042540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 10 monclient: shutdown 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 >> v1:172.21.15.121:6791/0 conn(0x7f11a009d150 legacy=0x7f11a01391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a640e640 1 -- 172.21.15.121:0/849918267 reap_dead start 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a640e640 1 -- 172.21.15.121:0/849918267 reap_dead start 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 shutdown_connections 2024-09-13T13:52:57.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 >> 172.21.15.121:0/849918267 conn(0x7f11a00ab940 msgr2=0x7f11a00aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:57.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 shutdown_connections 2024-09-13T13:52:57.973 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:57.936+0000 7f11a7e71640 1 -- 172.21.15.121:0/849918267 wait complete. 2024-09-13T13:52:57.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:57.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-13T13:52:57.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-13T13:52:58.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.069+0000 7f0d445e8640 10 monclient: get_monmap_and_config 2024-09-13T13:52:58.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.069+0000 7f0d445e8640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.069+0000 7f0d445e8640 10 monclient: monmap: 2024-09-13T13:52:58.071 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.071 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:58.071324+0000 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:58.071324+0000 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.072 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.073 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 1 Processor -- start 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 1 -- start start 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient: init 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient: _add_conns ranks=[7,4,2,8,5,6,0,1,3] 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient(hunting): picked mon.noname-i con 0x7f0d3c138bd0 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c138fb0 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c139420 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c09e100 con 0x7f0d3c139420 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c09db40 con 0x7f0d3c138fb0 2024-09-13T13:52:58.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c09de20 con 0x7f0d3c138bd0 2024-09-13T13:52:58.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.070+0000 7f0d445e8640 10 monclient(hunting): authenticate will time out at 857.700900s 2024-09-13T13:52:58.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.071+0000 7f0d41b83640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f0d3c138fb0 0x7f0d3c13df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33544/0 (socket says 172.21.15.121:33544) 2024-09-13T13:52:58.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.071+0000 7f0d42b85640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f0d3c139420 0x7f0d3c141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55008/0 (socket says 172.21.15.121:55008) 2024-09-13T13:52:58.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.071+0000 7f0d41b83640 1 -- 172.21.15.121:0/3566330473 learned_addr learned my addr 172.21.15.121:0/3566330473 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41b83640 1 -- 172.21.15.121:0/3566330473 >> v1:172.21.15.121:6791/0 conn(0x7f0d3c138fb0 legacy=0x7f0d3c13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d42b85640 1 -- 172.21.15.121:0/3566330473 >> v1:172.21.15.121:6790/0 conn(0x7f0d3c139420 legacy=0x7f0d3c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3673407348 0 0) 0x7f0d3c09db40 con 0x7f0d3c138fb0 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): my global_id is 4504 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d28003520 con 0x7f0d3c138fb0 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d42384640 1 -- 172.21.15.121:0/3566330473 >> v1:172.21.15.153:6792/0 conn(0x7f0d3c138bd0 legacy=0x7f0d3c0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488843030 0 0) 0x7f0d3c09e100 con 0x7f0d3c139420 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): my global_id is 4556 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c09db40 con 0x7f0d3c139420 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050663561 0 0) 0x7f0d3c09de20 con 0x7f0d3c138bd0 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient(hunting): my global_id is 104128 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c09e100 con 0x7f0d3c138bd0 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 459343934 0 0) 0x7f0d28003520 con 0x7f0d3c138fb0 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 >> v1:172.21.15.153:6792/0 conn(0x7f0d3c138bd0 legacy=0x7f0d3c0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 >> v1:172.21.15.121:6790/0 conn(0x7f0d3c139420 legacy=0x7f0d3c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 monclient: found mon.noname-c 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c0a5da0 con 0x7f0d3c138fb0 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f0d2c001ae0 con 0x7f0d3c138fb0 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d445e8640 5 monclient: authenticate success, global_id 4504 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d445e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: dump: 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.088 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.089 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d445e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d2c0048a0 con 0x7f0d3c138fb0 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 1 -- 172.21.15.121:0/3566330473 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f0d2c004e10 con 0x7f0d3c138fb0 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d445e8640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d445e8640 4 set_mon_vals no callback set 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.073+0000 7f0d41382640 10 monclient: dump: 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.090 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 10 monclient: shutdown 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 1 -- 172.21.15.121:0/3566330473 >> v1:172.21.15.121:6791/0 conn(0x7f0d3c138fb0 legacy=0x7f0d3c13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 1 -- 172.21.15.121:0/3566330473 shutdown_connections 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 1 -- 172.21.15.121:0/3566330473 >> 172.21.15.121:0/3566330473 conn(0x7f0d3c0ab800 msgr2=0x7f0d3c0abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 1 -- 172.21.15.121:0/3566330473 shutdown_connections 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 1 -- 172.21.15.121:0/3566330473 wait complete. 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.074+0000 7f0d445e8640 10 monclient: monmap: 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:58.092 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 1 Processor -- start 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 1 -- start start 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient: init 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient: _add_conns ranks=[0,1,3,4,2,5,8,7,6] 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c138bd0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient(hunting): picked mon.noname-f con 0x7f0d3c138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient(hunting): picked mon.noname-g con 0x7f0d3c139420 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c0a7ab0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c12e840 con 0x7f0d3c138fb0 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c130e80 con 0x7f0d3c139420 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d445e8640 10 monclient(hunting): authenticate will time out at 857.705857s 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d42384640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0d3c138bd0 0x7f0d3c12da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59052/0 (socket says 172.21.15.121:59052) 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.075+0000 7f0d42384640 1 -- 172.21.15.121:0/129466798 learned_addr learned my addr 172.21.15.121:0/129466798 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d42384640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6789/0 conn(0x7f0d3c138bd0 legacy=0x7f0d3c12da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860549068 0 0) 0x7f0d3c0a7ab0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): my global_id is 4635 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d20003520 con 0x7f0d3c138bd0 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d42b85640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.153:6790/0 conn(0x7f0d3c139420 legacy=0x7f0d3c1d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d41b83640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.153:6789/0 conn(0x7f0d3c138fb0 legacy=0x7f0d3c12e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 997098483 0 0) 0x7f0d20003520 con 0x7f0d3c138bd0 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d3c0a7ab0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f0d38003950 con 0x7f0d3c138bd0 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): dump: 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.095 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.096 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839449626 0 0) 0x7f0d3c130e80 con 0x7f0d3c139420 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): my global_id is 4494 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d20003520 con 0x7f0d3c139420 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634499498 0 0) 0x7f0d3c12e840 con 0x7f0d3c138fb0 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient(hunting): my global_id is 104113 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c130e80 con 0x7f0d3c138fb0 2024-09-13T13:52:58.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 898738621 0 0) 0x7f0d3c0a7ab0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.153:6790/0 conn(0x7f0d3c139420 legacy=0x7f0d3c1d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.153:6789/0 conn(0x7f0d3c138fb0 legacy=0x7f0d3c12e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 monclient: found mon.a 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c1301f0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 5 monclient: authenticate success, global_id 4635 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 10 monclient: _renew_subs 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c132d10 con 0x7f0d3c138bd0 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 10 monclient: _renew_subs 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1d7880 con 0x7f0d3c138bd0 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0d38001d20 con 0x7f0d3c138bd0 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f0d380041a0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d40b81640 4 set_mon_vals no callback set 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.078+0000 7f0d32ffd640 10 monclient: dump: 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.099 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.100 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.079+0000 7f0d445e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.079+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0d240065c0 con 0x7f0d24002860 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.079+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f0d38011ca0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.079+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f0d3804e3c0 con 0x7f0d3c138bd0 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.080+0000 7f0d42b85640 10 monclient: get_auth_request con 0x7f0d2003fbf0 auth_method 0 2024-09-13T13:52:58.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d41b83640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6791/0 conn(0x7f0d24002860 legacy=0x7f0d24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877748049 0 0) 0x7f0d240065c0 con 0x7f0d24002860 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 10 monclient(hunting): my global_id is 4513 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d3c0a7ab0 con 0x7f0d24002860 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.081+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2188706105 0 0) 0x7f0d3c0a7ab0 con 0x7f0d24002860 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d240065c0 con 0x7f0d24002860 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f0d2c001d30 con 0x7f0d24002860 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 10 monclient: dump: 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.102 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2639382590 0 0) 0x7f0d240065c0 con 0x7f0d24002860 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d2400a160 con 0x7f0d24002860 2024-09-13T13:52:58.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0d2c004b90 con 0x7f0d24002860 2024-09-13T13:52:58.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:58.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6791/0 conn(0x7f0d24002860 legacy=0x7f0d24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.082+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6791/0 conn(0x7f0d24002860 legacy=0x7f0d24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.088+0000 7f0d445e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:58.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.088+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f0d24009e70 con 0x7f0d24009130 2024-09-13T13:52:58.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d41b83640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6791/0 conn(0x7f0d24009130 legacy=0x7f0d240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1526718868 0 0) 0x7f0d24009e70 con 0x7f0d24009130 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d32ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d32ffd640 10 monclient(hunting): my global_id is 4522 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.090+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d240065c0 con 0x7f0d24009130 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2909659385 0 0) 0x7f0d240065c0 con 0x7f0d24009130 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0d24009e70 con 0x7f0d24009130 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f0d2c005490 con 0x7f0d24009130 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 10 monclient: dump: 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.106 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3453650147 0 0) 0x7f0d24009e70 con 0x7f0d24009130 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 --> v1:172.21.15.121:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d2400c3f0 con 0x7f0d24009130 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 <== mon.4 v1:172.21.15.121:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 1528946841) 0x7f0d2c007e40 con 0x7f0d24009130 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:58.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6791/0 conn(0x7f0d24009130 legacy=0x7f0d240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d32ffd640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6791/0 conn(0x7f0d24009130 legacy=0x7f0d240095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:58.108 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 101760, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.109 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.110 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:58.112 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.113 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:58.114 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.116 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.117 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.118 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.119 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.120 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.122 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.123 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:58.126 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr": [ 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:58.127 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.153:6812/1206688214 conn(0x7f0d2003fbf0 legacy=0x7f0d200420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 10 monclient: shutdown 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 >> v1:172.21.15.121:6789/0 conn(0x7f0d3c138bd0 legacy=0x7f0d3c12da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d42b85640 1 -- 172.21.15.121:0/129466798 reap_dead start 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d42b85640 1 -- 172.21.15.121:0/129466798 reap_dead start 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 shutdown_connections 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 >> 172.21.15.121:0/129466798 conn(0x7f0d3c0ab800 msgr2=0x7f0d3c12c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 shutdown_connections 2024-09-13T13:52:58.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.091+0000 7f0d445e8640 1 -- 172.21.15.121:0/129466798 wait complete. 2024-09-13T13:52:58.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:58.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-13T13:52:58.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.226+0000 7f8b33bf5640 10 monclient: get_monmap_and_config 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.226+0000 7f8b33bf5640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.226+0000 7f8b33bf5640 10 monclient: monmap: 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:58.228025+0000 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:58.228025+0000 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.228 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.229 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.226+0000 7f8b33bf5640 1 Processor -- start 2024-09-13T13:52:58.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 1 -- start start 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient: init 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient: _add_conns ranks=[5,2,4,1,6,3,8,0,7] 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient(hunting): picked mon.noname-h con 0x7f8b2c138bd0 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient(hunting): picked mon.noname-b con 0x7f8b2c138fb0 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient(hunting): picked mon.noname-c con 0x7f8b2c139420 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c09e100 con 0x7f8b2c138fb0 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c09db40 con 0x7f8b2c139420 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c09de20 con 0x7f8b2c138bd0 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b33bf5640 10 monclient(hunting): authenticate will time out at 857.857666s 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b31190640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f8b2c138fb0 0x7f8b2c13df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55024/0 (socket says 172.21.15.121:55024) 2024-09-13T13:52:58.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.227+0000 7f8b31190640 1 -- 172.21.15.121:0/557763452 learned_addr learned my addr 172.21.15.121:0/557763452 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.229+0000 7f8b32192640 1 -- 172.21.15.121:0/557763452 >> v1:172.21.15.121:6791/0 conn(0x7f8b2c139420 legacy=0x7f8b2c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b31190640 1 -- 172.21.15.121:0/557763452 >> v1:172.21.15.121:6790/0 conn(0x7f8b2c138fb0 legacy=0x7f8b2c13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b31991640 1 -- 172.21.15.121:0/557763452 >> v1:172.21.15.153:6791/0 conn(0x7f8b2c138bd0 legacy=0x7f8b2c0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2112441664 0 0) 0x7f8b2c09e100 con 0x7f8b2c138fb0 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): my global_id is 4565 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b14003520 con 0x7f8b2c138fb0 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494518645 0 0) 0x7f8b2c09db40 con 0x7f8b2c139420 2024-09-13T13:52:58.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): my global_id is 4531 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2c09e100 con 0x7f8b2c139420 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 593608252 0 0) 0x7f8b2c09de20 con 0x7f8b2c138bd0 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient(hunting): my global_id is 4550 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2c09db40 con 0x7f8b2c138bd0 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1195624310 0 0) 0x7f8b14003520 con 0x7f8b2c138fb0 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 >> v1:172.21.15.153:6791/0 conn(0x7f8b2c138bd0 legacy=0x7f8b2c0a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 >> v1:172.21.15.121:6791/0 conn(0x7f8b2c139420 legacy=0x7f8b2c141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 monclient: found mon.noname-b 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c0a5da0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8b20001ae0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b33bf5640 5 monclient: authenticate success, global_id 4565 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b33bf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:52:58.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: dump: 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.239 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b33bf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b200048a0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 1 -- 172.21.15.121:0/557763452 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8b20004e10 con 0x7f8b2c138fb0 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b33bf5640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b33bf5640 4 set_mon_vals no callback set 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.230+0000 7f8b3098f640 10 monclient: dump: 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.240 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 10 monclient: shutdown 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 1 -- 172.21.15.121:0/557763452 >> v1:172.21.15.121:6790/0 conn(0x7f8b2c138fb0 legacy=0x7f8b2c13df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 1 -- 172.21.15.121:0/557763452 shutdown_connections 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 1 -- 172.21.15.121:0/557763452 >> 172.21.15.121:0/557763452 conn(0x7f8b2c0ab800 msgr2=0x7f8b2c0abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 1 -- 172.21.15.121:0/557763452 shutdown_connections 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 1 -- 172.21.15.121:0/557763452 wait complete. 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.231+0000 7f8b33bf5640 10 monclient: monmap: 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:58.242 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 1 Processor -- start 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 1 -- start start 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient: init 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient: _add_conns ranks=[3,6,1,8,7,4,2,0,5] 2024-09-13T13:52:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient(hunting): picked mon.noname-g con 0x7f8b2c138bd0 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient(hunting): picked mon.noname-d con 0x7f8b2c138fb0 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient(hunting): picked mon.noname-f con 0x7f8b2c139420 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c0a4ce0 con 0x7f8b2c139420 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c0a4130 con 0x7f8b2c138bd0 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c13a9b0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b33bf5640 10 monclient(hunting): authenticate will time out at 857.862633s 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b31190640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f8b2c138fb0 0x7f8b2c0a3a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51404/0 (socket says 172.21.15.121:51404) 2024-09-13T13:52:58.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.232+0000 7f8b31190640 1 -- 172.21.15.121:0/73454172 learned_addr learned my addr 172.21.15.121:0/73454172 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b31190640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b2c138fb0 legacy=0x7f8b2c0a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360318075 0 0) 0x7f8b2c13a9b0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b2a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b2a7fc640 10 monclient(hunting): my global_id is 4605 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.234+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b18003520 con 0x7f8b2c138fb0 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b31991640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.153:6790/0 conn(0x7f8b2c138bd0 legacy=0x7f8b2c0a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b32192640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.153:6789/0 conn(0x7f8b2c139420 legacy=0x7f8b2c1d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2864081431 0 0) 0x7f8b18003520 con 0x7f8b2c138fb0 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b2c13a9b0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8b20004d90 con 0x7f8b2c138fb0 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.250 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1760663501 0 0) 0x7f8b2c0a4ce0 con 0x7f8b2c139420 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): my global_id is 104122 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b18003520 con 0x7f8b2c139420 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1173648952 0 0) 0x7f8b2c0a4130 con 0x7f8b2c138bd0 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient(hunting): my global_id is 4503 2024-09-13T13:52:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2c0a4ce0 con 0x7f8b2c138bd0 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2182468625 0 0) 0x7f8b2c13a9b0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.153:6790/0 conn(0x7f8b2c138bd0 legacy=0x7f8b2c0a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.153:6789/0 conn(0x7f8b2c139420 legacy=0x7f8b2c1d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 monclient: found mon.d 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c13b250 con 0x7f8b2c138fb0 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 5 monclient: authenticate success, global_id 4605 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 10 monclient: _renew_subs 2024-09-13T13:52:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2c13b670 con 0x7f8b2c138fb0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 10 monclient: _renew_subs 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b2c139a50 con 0x7f8b2c138fb0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b20001d30 con 0x7f8b2c138fb0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8b200048a0 con 0x7f8b2c138fb0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2bfff640 4 set_mon_vals no callback set 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.235+0000 7f8b2a7fc640 10 monclient: dump: 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.236+0000 7f8b33bf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.236+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b0c0065c0 con 0x7f8b0c002860 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.236+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f8b20011c20 con 0x7f8b2c138fb0 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.236+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f8b2004e280 con 0x7f8b2c138fb0 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.236+0000 7f8b32192640 10 monclient: get_auth_request con 0x7f8b1803feb0 auth_method 0 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b31991640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b0c002860 legacy=0x7f8b0c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090353299 0 0) 0x7f8b0c0065c0 con 0x7f8b0c002860 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient(hunting): my global_id is 4614 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b2c13a9b0 con 0x7f8b0c002860 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3803239324 0 0) 0x7f8b2c13a9b0 con 0x7f8b0c002860 2024-09-13T13:52:58.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b0c0065c0 con 0x7f8b0c002860 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8b24001c60 con 0x7f8b0c002860 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 10 monclient: dump: 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.256 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3190339858 0 0) 0x7f8b0c0065c0 con 0x7f8b0c002860 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.238+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b0c00a160 con 0x7f8b0c002860 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.239+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8b24006870 con 0x7f8b0c002860 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.239+0000 7f8b2a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.239+0000 7f8b2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.239+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b0c002860 legacy=0x7f8b0c004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.239+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b0c002860 legacy=0x7f8b0c004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.245+0000 7f8b33bf5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.245+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8b0c009e50 con 0x7f8b0c009110 2024-09-13T13:52:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b31991640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b0c009110 legacy=0x7f8b0c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 250638042 0 0) 0x7f8b0c009e50 con 0x7f8b0c009110 2024-09-13T13:52:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b2a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b2a7fc640 10 monclient(hunting): my global_id is 4623 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.247+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b0c0065c0 con 0x7f8b0c009110 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2415382103 0 0) 0x7f8b0c0065c0 con 0x7f8b0c009110 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b0c009e50 con 0x7f8b0c009110 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8b24003d40 con 0x7f8b0c009110 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 10 monclient: dump: 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1811389619 0 0) 0x7f8b0c009e50 con 0x7f8b0c009110 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 --> v1:172.21.15.121:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8b0c00c3f0 con 0x7f8b0c009110 2024-09-13T13:52:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 <== mon.6 v1:172.21.15.121:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7561 (unknown 0 0 945762595) 0x7f8b24006ae0 con 0x7f8b0c009110 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b0c009110 legacy=0x7f8b0c0095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b2a7fc640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b0c009110 legacy=0x7f8b0c0095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "leader", 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:58.262 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 101921, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:58.263 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:58.264 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:58.266 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.267 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.268 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.269 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.270 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.271 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.272 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.273 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.274 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.275 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:58.276 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.277 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.278 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.279 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.280 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 3 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:58.281 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.153:6812/1206688214 conn(0x7f8b1803feb0 legacy=0x7f8b18042370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 10 monclient: shutdown 2024-09-13T13:52:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 >> v1:172.21.15.121:6792/0 conn(0x7f8b2c138fb0 legacy=0x7f8b2c0a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b32192640 1 -- 172.21.15.121:0/73454172 reap_dead start 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b32192640 1 -- 172.21.15.121:0/73454172 reap_dead start 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 shutdown_connections 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 >> 172.21.15.121:0/73454172 conn(0x7f8b2c0ab800 msgr2=0x7f8b2c137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 shutdown_connections 2024-09-13T13:52:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.248+0000 7f8b33bf5640 1 -- 172.21.15.121:0/73454172 wait complete. 2024-09-13T13:52:58.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:58.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-13T13:52:58.284 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.380+0000 7f78c84bd640 10 monclient: get_monmap_and_config 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.380+0000 7f78c84bd640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.380+0000 7f78c84bd640 10 monclient: monmap: 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:58.382510+0000 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:58.382510+0000 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.383 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.381+0000 7f78c84bd640 1 Processor -- start 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.381+0000 7f78c84bd640 1 -- start start 2024-09-13T13:52:58.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.381+0000 7f78c84bd640 10 monclient: init 2024-09-13T13:52:58.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient: _add_conns ranks=[2,8,5,4,7,1,6,3,0] 2024-09-13T13:52:58.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient(hunting): picked mon.noname-b con 0x7f78c00a6160 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient(hunting): picked mon.noname-e con 0x7f78c009d150 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient(hunting): picked mon.noname-h con 0x7f78c00a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c009e210 con 0x7f78c00a6160 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c00a7ef0 con 0x7f78c00a6c50 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c009d970 con 0x7f78c009d150 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c84bd640 10 monclient(hunting): authenticate will time out at 858.012362s 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c6259640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f78c00a6160 0x7f78c00a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55026/0 (socket says 172.21.15.121:55026) 2024-09-13T13:52:58.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.382+0000 7f78c6259640 1 -- 172.21.15.121:0/1975000745 learned_addr learned my addr 172.21.15.121:0/1975000745 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.384+0000 7f78c6259640 1 -- 172.21.15.121:0/1975000745 >> v1:172.21.15.121:6790/0 conn(0x7f78c00a6160 legacy=0x7f78c00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.384+0000 7f78c6a5a640 1 -- 172.21.15.121:0/1975000745 >> v1:172.21.15.153:6791/0 conn(0x7f78c00a6c50 legacy=0x7f78c0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.384+0000 7f78c5a58640 1 -- 172.21.15.121:0/1975000745 >> v1:172.21.15.121:6793/0 conn(0x7f78c009d150 legacy=0x7f78c013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.390+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724720239 0 0) 0x7f78c00a7ef0 con 0x7f78c00a6c50 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.390+0000 7f78c5257640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.390+0000 7f78c5257640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.390+0000 7f78c5257640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.390+0000 7f78c5257640 10 monclient(hunting): my global_id is 4559 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.390+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78ac003520 con 0x7f78c00a6c50 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 108773241 0 0) 0x7f78ac003520 con 0x7f78c00a6c50 2024-09-13T13:52:58.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 >> v1:172.21.15.121:6793/0 conn(0x7f78c009d150 legacy=0x7f78c013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 >> v1:172.21.15.121:6790/0 conn(0x7f78c00a6160 legacy=0x7f78c00a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 monclient: found mon.noname-h 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.153:6791/0 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c009e630 con 0x7f78c00a6c50 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78b8001eb0 con 0x7f78c00a6c50 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: dump: 2024-09-13T13:52:58.396 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.397 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c84bd640 5 monclient: authenticate success, global_id 4559 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c84bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 <== mon.5 v1:172.21.15.153:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78b8003990 con 0x7f78c00a6c50 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 1 -- 172.21.15.121:0/1975000745 <== mon.5 v1:172.21.15.153:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78b8003f00 con 0x7f78c00a6c50 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c5257640 10 monclient: dump: 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.398 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c84bd640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.391+0000 7f78c84bd640 4 set_mon_vals no callback set 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 10 monclient: shutdown 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 1 -- 172.21.15.121:0/1975000745 >> v1:172.21.15.153:6791/0 conn(0x7f78c00a6c50 legacy=0x7f78c0141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 1 -- 172.21.15.121:0/1975000745 shutdown_connections 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 1 -- 172.21.15.121:0/1975000745 >> 172.21.15.121:0/1975000745 conn(0x7f78c00ab940 msgr2=0x7f78c00abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 1 -- 172.21.15.121:0/1975000745 shutdown_connections 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.392+0000 7f78c84bd640 1 -- 172.21.15.121:0/1975000745 wait complete. 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.393+0000 7f78c84bd640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.393+0000 7f78c84bd640 10 monclient: monmap: 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.400 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.393+0000 7f78c84bd640 1 Processor -- start 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.393+0000 7f78c84bd640 1 -- start start 2024-09-13T13:52:58.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.393+0000 7f78c84bd640 10 monclient: init 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient: _add_conns ranks=[5,8,4,3,1,2,0,6,7] 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient(hunting): picked mon.noname-h con 0x7f78c009d150 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient(hunting): picked mon.noname-e con 0x7f78c00a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient(hunting): picked mon.noname-c con 0x7f78c00a6c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c00a4670 con 0x7f78c00a6c50 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c0124800 con 0x7f78c009d150 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c0126b60 con 0x7f78c00a6160 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c84bd640 10 monclient(hunting): authenticate will time out at 858.024175s 2024-09-13T13:52:58.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c5a58640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f78c00a6160 0x7f78c01d2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:47980/0 (socket says 172.21.15.121:47980) 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c5a58640 1 -- 172.21.15.121:0/3709828966 learned_addr learned my addr 172.21.15.121:0/3709828966 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c6a5a640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f78c00a6c50 0x7f78c01d63d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33582/0 (socket says 172.21.15.121:33582) 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c6259640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.153:6791/0 conn(0x7f78c009d150 legacy=0x7f78c01391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c6259640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.153:6791/0 conn(0x7f78c009d150 legacy=0x7f78c01391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c6259640 1 --1- 172.21.15.121:0/3709828966 >> v1:172.21.15.153:6791/0 conn(0x7f78c009d150 0x7f78c01391a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78c6259640 1 --1- 172.21.15.121:0/3709828966 >> v1:172.21.15.153:6791/0 conn(0x7f78c009d150 0x7f78c01391a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.394+0000 7f78b2ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6791/0 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78c5a58640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78c00a6160 legacy=0x7f78c01d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78c6a5a640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6791/0 conn(0x7f78c00a6c50 legacy=0x7f78c01d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883833711 0 0) 0x7f78c0126b60 con 0x7f78c00a6160 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): my global_id is 4598 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a4003520 con 0x7f78c00a6160 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2757721650 0 0) 0x7f78c00a4670 con 0x7f78c00a6c50 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): my global_id is 4540 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c0126b60 con 0x7f78c00a6c50 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2721854647 0 0) 0x7f78a4003520 con 0x7f78c00a6160 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78c00a4670 con 0x7f78c00a6160 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78bc003610 con 0x7f78c00a6160 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): dump: 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.410 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2787100259 0 0) 0x7f78c0126b60 con 0x7f78c00a6c50 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78a4003520 con 0x7f78c00a6c50 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78b8003e80 con 0x7f78c00a6c50 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 10 monclient(hunting): dump: 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.411 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.412 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.396+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3084754882 0 0) 0x7f78c00a4670 con 0x7f78c00a6160 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.153:6791/0 conn(0x7f78c009d150 legacy=0x7f78c01391a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6791/0 conn(0x7f78c00a6c50 legacy=0x7f78c01d63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 monclient: found mon.e 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c0125360 con 0x7f78c00a6160 2024-09-13T13:52:58.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 5 monclient: authenticate success, global_id 4598 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 10 monclient: _renew_subs 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78c01398b0 con 0x7f78c00a6160 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 10 monclient: _renew_subs 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f78c01cdd70 con 0x7f78c00a6160 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78bc003bc0 con 0x7f78c00a6160 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78bc0040b0 con 0x7f78c00a6160 2024-09-13T13:52:58.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78c4a56640 4 set_mon_vals no callback set 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 10 monclient: dump: 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.415 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.397+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f78bc004250 con 0x7f78c00a6160 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.398+0000 7f78c84bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.398+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78a80065c0 con 0x7f78a8002860 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.398+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f78bc04e320 con 0x7f78c00a6160 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.398+0000 7f78c6259640 10 monclient: get_auth_request con 0x7f78a4040120 auth_method 0 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78c6a5a640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78a8002860 legacy=0x7f78a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195788545 0 0) 0x7f78a80065c0 con 0x7f78a8002860 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient(hunting): my global_id is 4607 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c00a4670 con 0x7f78a8002860 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 760616559 0 0) 0x7f78c00a4670 con 0x7f78a8002860 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78a80065c0 con 0x7f78a8002860 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78b8001950 con 0x7f78a8002860 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 10 monclient: dump: 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.417 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1632818749 0 0) 0x7f78a80065c0 con 0x7f78a8002860 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.400+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78a800a160 con 0x7f78a8002860 2024-09-13T13:52:58.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.401+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f78b8005b40 con 0x7f78a8002860 2024-09-13T13:52:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.401+0000 7f78b2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.401+0000 7f78b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.401+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78a8002860 legacy=0x7f78a8004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.401+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78a8002860 legacy=0x7f78a8004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.407+0000 7f78c84bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.407+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78a8009e70 con 0x7f78a8009130 2024-09-13T13:52:58.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78c6a5a640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78a8009130 legacy=0x7f78a80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2386675471 0 0) 0x7f78a8009e70 con 0x7f78a8009130 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient(hunting): my global_id is 4616 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78a80065c0 con 0x7f78a8009130 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1420414606 0 0) 0x7f78a80065c0 con 0x7f78a8009130 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78a8009e70 con 0x7f78a8009130 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f78b8005cf0 con 0x7f78a8009130 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.409+0000 7f78b2ffd640 10 monclient: dump: 2024-09-13T13:52:58.421 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.422 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 6884841 0 0) 0x7f78a8009e70 con 0x7f78a8009130 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 --> v1:172.21.15.121:6793/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78a800c410 con 0x7f78a8009130 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 <== mon.8 v1:172.21.15.121:6793/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7559 (unknown 0 0 2736475984) 0x7f78b8005fd0 con 0x7f78a8009130 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78a8009130 legacy=0x7f78a80095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78b2ffd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78a8009130 legacy=0x7f78a80095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.423 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 102068, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:58.424 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.425 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.426 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:58.427 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.428 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.429 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.430 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.431 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.432 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.433 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.434 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.435 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.437 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:58.439 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.440 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:58.441 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.153:6812/1206688214 conn(0x7f78a4040120 legacy=0x7f78a40425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 10 monclient: shutdown 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 >> v1:172.21.15.121:6793/0 conn(0x7f78c00a6160 legacy=0x7f78c01d2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c6a5a640 1 -- 172.21.15.121:0/3709828966 reap_dead start 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 shutdown_connections 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 >> 172.21.15.121:0/3709828966 conn(0x7f78c00ab940 msgr2=0x7f78c009fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 shutdown_connections 2024-09-13T13:52:58.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.410+0000 7f78c84bd640 1 -- 172.21.15.121:0/3709828966 wait complete. 2024-09-13T13:52:58.444 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:58.444 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-13T13:52:58.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-13T13:52:58.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.540+0000 7f7938564640 10 monclient: get_monmap_and_config 2024-09-13T13:52:58.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.540+0000 7f7938564640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.540+0000 7f7938564640 10 monclient: monmap: 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:58.542568+0000 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:58.542568+0000 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.544 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.541+0000 7f7938564640 1 Processor -- start 2024-09-13T13:52:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.541+0000 7f7938564640 1 -- start start 2024-09-13T13:52:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.541+0000 7f7938564640 10 monclient: init 2024-09-13T13:52:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient: _add_conns ranks=[4,1,2,8,7,3,0,6,5] 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient(hunting): picked mon.noname-c con 0x7f7930138bd0 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient(hunting): picked mon.noname-f con 0x7f7930138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient(hunting): picked mon.noname-b con 0x7f7930139420 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793009e100 con 0x7f7930138fb0 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793009db40 con 0x7f7930139420 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793009de20 con 0x7f7930138bd0 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7938564640 10 monclient(hunting): authenticate will time out at 858.172177s 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7936b01640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f7930139420 0x7f7930141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55032/0 (socket says 172.21.15.121:55032) 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7936300640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f7930138bd0 0x7f79300a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33592/0 (socket says 172.21.15.121:33592) 2024-09-13T13:52:58.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.542+0000 7f7936b01640 1 -- 172.21.15.121:0/1035646572 learned_addr learned my addr 172.21.15.121:0/1035646572 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f7936b01640 1 -- 172.21.15.121:0/1035646572 >> v1:172.21.15.121:6790/0 conn(0x7f7930139420 legacy=0x7f7930141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f7936300640 1 -- 172.21.15.121:0/1035646572 >> v1:172.21.15.121:6791/0 conn(0x7f7930138bd0 legacy=0x7f79300a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2998042071 0 0) 0x7f793009db40 con 0x7f7930139420 2024-09-13T13:52:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): my global_id is 4574 2024-09-13T13:52:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f7935aff640 1 -- 172.21.15.121:0/1035646572 >> v1:172.21.15.153:6789/0 conn(0x7f7930138fb0 legacy=0x7f793013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f791c003520 con 0x7f7930139420 2024-09-13T13:52:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668559954 0 0) 0x7f793009de20 con 0x7f7930138bd0 2024-09-13T13:52:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): my global_id is 4549 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f793009db40 con 0x7f7930138bd0 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1677745118 0 0) 0x7f793009e100 con 0x7f7930138fb0 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient(hunting): my global_id is 104131 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f793009de20 con 0x7f7930138fb0 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1575627058 0 0) 0x7f791c003520 con 0x7f7930139420 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 >> v1:172.21.15.121:6791/0 conn(0x7f7930138bd0 legacy=0x7f79300a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 >> v1:172.21.15.153:6789/0 conn(0x7f7930138fb0 legacy=0x7f793013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 monclient: found mon.noname-b 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79300a5da0 con 0x7f7930139420 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7928001ae0 con 0x7f7930139420 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f7938564640 5 monclient: authenticate success, global_id 4574 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f7938564640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:52:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f79352fe640 10 monclient: dump: 2024-09-13T13:52:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.555 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.544+0000 7f7938564640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f79280048a0 con 0x7f7930139420 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f79352fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f79352fe640 1 -- 172.21.15.121:0/1035646572 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7928004e10 con 0x7f7930139420 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 4 set_mon_vals no callback set 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f79352fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f79352fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f79352fe640 10 monclient: dump: 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.557 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 10 monclient: shutdown 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 1 -- 172.21.15.121:0/1035646572 >> v1:172.21.15.121:6790/0 conn(0x7f7930139420 legacy=0x7f7930141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 1 -- 172.21.15.121:0/1035646572 shutdown_connections 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 1 -- 172.21.15.121:0/1035646572 >> 172.21.15.121:0/1035646572 conn(0x7f79300ab800 msgr2=0x7f79300abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 1 -- 172.21.15.121:0/1035646572 shutdown_connections 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.545+0000 7f7938564640 1 -- 172.21.15.121:0/1035646572 wait complete. 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient: monmap: 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.559 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 1 Processor -- start 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 1 -- start start 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient: init 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient: _add_conns ranks=[6,1,7,2,5,4,0,3,8] 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient(hunting): picked mon.noname-d con 0x7f7930138bd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient(hunting): picked mon.noname-f con 0x7f7930138fb0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 10 monclient(hunting): picked mon.noname-i con 0x7f7930139420 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.546+0000 7f7938564640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79300a5580 con 0x7f7930138fb0 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.547+0000 7f7938564640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79300a7680 con 0x7f7930138bd0 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.547+0000 7f7938564640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793013ab70 con 0x7f7930139420 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.547+0000 7f7938564640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.547+0000 7f7938564640 10 monclient(hunting): authenticate will time out at 858.177038s 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.547+0000 7f7936300640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f7930138bd0 0x7f79300a3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51424/0 (socket says 172.21.15.121:51424) 2024-09-13T13:52:58.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.547+0000 7f7936300640 1 -- 172.21.15.121:0/820686313 learned_addr learned my addr 172.21.15.121:0/820686313 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7936300640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.121:6792/0 conn(0x7f7930138bd0 legacy=0x7f79300a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 181731340 0 0) 0x7f792c002060 con 0x7f7930138bd0 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): my global_id is 4632 2024-09-13T13:52:58.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7914003520 con 0x7f7930138bd0 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7936b01640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6792/0 conn(0x7f7930139420 legacy=0x7f79301d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7935aff640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7930138fb0 legacy=0x7f79300a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1454781041 0 0) 0x7f7914003520 con 0x7f7930138bd0 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f792c002060 con 0x7f7930138bd0 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f792c001cb0 con 0x7f7930138bd0 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): dump: 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.569 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 435003260 0 0) 0x7f793013ab70 con 0x7f7930139420 2024-09-13T13:52:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): my global_id is 104137 2024-09-13T13:52:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7914003520 con 0x7f7930139420 2024-09-13T13:52:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1739518777 0 0) 0x7f79300a5580 con 0x7f7930138fb0 2024-09-13T13:52:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient(hunting): my global_id is 104140 2024-09-13T13:52:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f793013ab70 con 0x7f7930138fb0 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2122666546 0 0) 0x7f792c002060 con 0x7f7930138bd0 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6792/0 conn(0x7f7930139420 legacy=0x7f79301d5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7930138fb0 legacy=0x7f79300a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 monclient: found mon.d 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793013ae50 con 0x7f7930138bd0 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 5 monclient: authenticate success, global_id 4632 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 10 monclient: _renew_subs 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f793013b410 con 0x7f7930138bd0 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 10 monclient: _renew_subs 2024-09-13T13:52:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f793013b130 con 0x7f7930138bd0 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f792c005150 con 0x7f7930138bd0 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f792c005640 con 0x7f7930138bd0 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7934afd640 4 set_mon_vals no callback set 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.549+0000 7f7926ffd640 10 monclient: dump: 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.576 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.550+0000 7f7938564640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.550+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f79180065c0 con 0x7f7918002860 2024-09-13T13:52:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.550+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f792c006910 con 0x7f7930138bd0 2024-09-13T13:52:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.551+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f792c04e840 con 0x7f7930138bd0 2024-09-13T13:52:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.551+0000 7f7936b01640 10 monclient: get_auth_request con 0x7f791403fd10 auth_method 0 2024-09-13T13:52:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.552+0000 7f7935aff640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7918002860 legacy=0x7f7918004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231734854 0 0) 0x7f79180065c0 con 0x7f7918002860 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient(hunting): my global_id is 104149 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f792c002060 con 0x7f7918002860 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3826033196 0 0) 0x7f792c002060 con 0x7f7918002860 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f79180065c0 con 0x7f7918002860 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7920001c60 con 0x7f7918002860 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 10 monclient: dump: 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.580 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 226791141 0 0) 0x7f79180065c0 con 0x7f7918002860 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.553+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f791800a160 con 0x7f7918002860 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.554+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f7920006850 con 0x7f7918002860 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.554+0000 7f7926ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.554+0000 7f7926ffd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.554+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7918002860 legacy=0x7f7918004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.554+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7918002860 legacy=0x7f7918004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.560+0000 7f7938564640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:58.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.560+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f7918009e50 con 0x7f7918009110 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7935aff640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7918009110 legacy=0x7f79180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685136320 0 0) 0x7f7918009e50 con 0x7f7918009110 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient(hunting): my global_id is 104158 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79180065c0 con 0x7f7918009110 2024-09-13T13:52:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1842112713 0 0) 0x7f79180065c0 con 0x7f7918009110 2024-09-13T13:52:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7918009e50 con 0x7f7918009110 2024-09-13T13:52:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7920003e00 con 0x7f7918009110 2024-09-13T13:52:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:52:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 10 monclient: dump: 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3980808822 0 0) 0x7f7918009e50 con 0x7f7918009110 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.562+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 --> v1:172.21.15.153:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f791800c3a0 con 0x7f7918009110 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 <== mon.1 v1:172.21.15.153:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7558 (unknown 0 0 4209110898) 0x7f79200040d0 con 0x7f7918009110 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7926ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7926ffd640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7918009110 legacy=0x7f79180095a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7926ffd640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6789/0 conn(0x7f7918009110 legacy=0x7f79180095a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.587 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:58.588 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.588 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.588 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:58.588 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:58.588 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 21839, 2024-09-13T13:52:58.588 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:58.589 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:58.589 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:58.589 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:58.589 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:58.589 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:58.590 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:58.590 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:58.590 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:58.590 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:58.590 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.590 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 1, 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.591 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.592 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:58.593 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:58.594 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.595 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.596 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.597 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:58.599 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.600 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:58.601 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.602 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.603 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.604 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.605 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.606 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.607 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.608 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.609 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:58.610 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:58.611 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:58.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.153:6812/1206688214 conn(0x7f791403fd10 legacy=0x7f79140421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 10 monclient: shutdown 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 >> v1:172.21.15.121:6792/0 conn(0x7f7930138bd0 legacy=0x7f79300a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7936b01640 1 -- 172.21.15.121:0/820686313 reap_dead start 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7936b01640 1 -- 172.21.15.121:0/820686313 reap_dead start 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 shutdown_connections 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 >> 172.21.15.121:0/820686313 conn(0x7f79300ab800 msgr2=0x7f7930137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 shutdown_connections 2024-09-13T13:52:58.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.563+0000 7f7938564640 1 -- 172.21.15.121:0/820686313 wait complete. 2024-09-13T13:52:58.613 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:58.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-13T13:52:58.614 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-13T13:52:58.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.711+0000 7f964ffda640 10 monclient: get_monmap_and_config 2024-09-13T13:52:58.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.711+0000 7f964ffda640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.711+0000 7f964ffda640 10 monclient: monmap: 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:58.713219+0000 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:58.713219+0000 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.714 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.715 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.715 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.715 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.715 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 1 Processor -- start 2024-09-13T13:52:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 1 -- start start 2024-09-13T13:52:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient: init 2024-09-13T13:52:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient: _add_conns ranks=[3,0,8,7,1,6,5,2,4] 2024-09-13T13:52:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient(hunting): picked mon.noname-g con 0x7f9648138bd0 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:58.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient(hunting): picked mon.noname-a con 0x7f9648138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient(hunting): picked mon.noname-e con 0x7f9648139420 addr v1:172.21.15.121:6793/0 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964809e100 con 0x7f9648138fb0 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964809db40 con 0x7f9648138bd0 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f964809de20 con 0x7f9648139420 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964ffda640 10 monclient(hunting): authenticate will time out at 858.342889s 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964e577640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f9648139420 0x7f9648141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:48006/0 (socket says 172.21.15.121:48006) 2024-09-13T13:52:58.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.712+0000 7f964e577640 1 -- 172.21.15.121:0/19638006 learned_addr learned my addr 172.21.15.121:0/19638006 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964e577640 1 -- 172.21.15.121:0/19638006 >> v1:172.21.15.121:6793/0 conn(0x7f9648139420 legacy=0x7f9648141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964d575640 1 -- 172.21.15.121:0/19638006 >> v1:172.21.15.121:6789/0 conn(0x7f9648138fb0 legacy=0x7f964813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135008685 0 0) 0x7f964809de20 con 0x7f9648139420 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): my global_id is 4625 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9634003520 con 0x7f9648139420 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174846929 0 0) 0x7f964809e100 con 0x7f9648138fb0 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): my global_id is 4644 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964809de20 con 0x7f9648138fb0 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964dd76640 1 -- 172.21.15.121:0/19638006 >> v1:172.21.15.153:6790/0 conn(0x7f9648138bd0 legacy=0x7f96480a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417034706 0 0) 0x7f964809db40 con 0x7f9648138bd0 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient(hunting): my global_id is 4512 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964809e100 con 0x7f9648138bd0 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2339593939 0 0) 0x7f964809de20 con 0x7f9648138fb0 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 >> v1:172.21.15.121:6793/0 conn(0x7f9648139420 legacy=0x7f9648141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 >> v1:172.21.15.153:6790/0 conn(0x7f9648138bd0 legacy=0x7f96480a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 monclient: found mon.noname-a 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96480a5da0 con 0x7f9648138fb0 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9638001a90 con 0x7f9648138fb0 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964ffda640 5 monclient: authenticate success, global_id 4644 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964ffda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: dump: 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.730 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964ffda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9638001ce0 con 0x7f9648138fb0 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 1 -- 172.21.15.121:0/19638006 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9638003950 con 0x7f9648138fb0 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964ffda640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964ffda640 4 set_mon_vals no callback set 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.715+0000 7f964cd74640 10 monclient: dump: 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.732 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 10 monclient: shutdown 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 1 -- 172.21.15.121:0/19638006 >> v1:172.21.15.121:6789/0 conn(0x7f9648138fb0 legacy=0x7f964813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 1 -- 172.21.15.121:0/19638006 shutdown_connections 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 1 -- 172.21.15.121:0/19638006 >> 172.21.15.121:0/19638006 conn(0x7f96480ab800 msgr2=0x7f96480abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 1 -- 172.21.15.121:0/19638006 shutdown_connections 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 1 -- 172.21.15.121:0/19638006 wait complete. 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.716+0000 7f964ffda640 10 monclient: monmap: 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:58.734 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 1 Processor -- start 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 1 -- start start 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient: init 2024-09-13T13:52:58.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient: _add_conns ranks=[6,2,3,4,5,8,1,0,7] 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient(hunting): picked mon.noname-d con 0x7f9648138bd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient(hunting): picked mon.noname-b con 0x7f9648138fb0 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient(hunting): picked mon.noname-g con 0x7f9648139420 addr v1:172.21.15.153:6790/0 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96480a58b0 con 0x7f9648138fb0 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9648132290 con 0x7f9648139420 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9648132570 con 0x7f9648138bd0 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964ffda640 10 monclient(hunting): authenticate will time out at 858.347658s 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964dd76640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 0x7f9648141cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51436/0 (socket says 172.21.15.121:51436) 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964d575640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f9648138fb0 0x7f96481d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55036/0 (socket says 172.21.15.121:55036) 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.717+0000 7f964dd76640 1 -- 172.21.15.121:0/2780657991 learned_addr learned my addr 172.21.15.121:0/2780657991 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.719+0000 7f964dd76640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 legacy=0x7f9648141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.719+0000 7f964d575640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6790/0 conn(0x7f9648138fb0 legacy=0x7f96481d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.719+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802267871 0 0) 0x7f9648132570 con 0x7f9648138bd0 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): my global_id is 4641 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964e577640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9648139420 legacy=0x7f96481d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f962c003520 con 0x7f9648138bd0 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659899598 0 0) 0x7f96480a58b0 con 0x7f9648138fb0 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): my global_id is 4583 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9648132570 con 0x7f9648138fb0 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997214776 0 0) 0x7f9648132290 con 0x7f9648139420 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): my global_id is 4521 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96480a58b0 con 0x7f9648139420 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1636373478 0 0) 0x7f962c003520 con 0x7f9648138bd0 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9648132290 con 0x7f9648138bd0 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9644003610 con 0x7f9648138bd0 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.739 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2062008626 0 0) 0x7f9648132570 con 0x7f9648138fb0 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f962c003520 con 0x7f9648138fb0 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964dd76640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 legacy=0x7f9648141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964dd76640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 legacy=0x7f9648141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9638003950 con 0x7f9648138fb0 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964dd76640 1 --1- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 0x7f9648141cd0 :-1 s=OPENED pgs=144 cs=1 l=1).handle_message read tag failed 2024-09-13T13:52:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964dd76640 1 --1- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 0x7f9648141cd0 :-1 s=OPENED pgs=144 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.741 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.121:6792/0 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1433909928 0 0) 0x7f96480a58b0 con 0x7f9648139420 2024-09-13T13:52:58.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9648132570 con 0x7f9648139420 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f96400020b0 con 0x7f9648139420 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.743 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 750260839 0 0) 0x7f962c003520 con 0x7f9648138fb0 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6792/0 conn(0x7f9648138bd0 legacy=0x7f9648141cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9648139420 legacy=0x7f96481d68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 monclient: found mon.b 2024-09-13T13:52:58.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9648130850 con 0x7f9648138fb0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 5 monclient: authenticate success, global_id 4583 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 10 monclient: _renew_subs 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9648131f60 con 0x7f9648138fb0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 10 monclient: _renew_subs 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f964812fe70 con 0x7f9648138fb0 2024-09-13T13:52:58.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9638001ab0 con 0x7f9648138fb0 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9638001ce0 con 0x7f9648138fb0 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963ffff640 4 set_mon_vals no callback set 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.720+0000 7f963e7fc640 10 monclient: dump: 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.746 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.721+0000 7f964ffda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.721+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f96300065c0 con 0x7f9630002860 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.721+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f96380042c0 con 0x7f9648138fb0 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.721+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f963804e0a0 con 0x7f9648138fb0 2024-09-13T13:52:58.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.722+0000 7f964e577640 10 monclient: get_auth_request con 0x7f962c0406d0 auth_method 0 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f964dd76640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9630002860 legacy=0x7f9630004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375181114 0 0) 0x7f96300065c0 con 0x7f9630002860 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f963e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f963e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f963e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f963e7fc640 10 monclient(hunting): my global_id is 4530 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.723+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f962c003520 con 0x7f9630002860 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2528566110 0 0) 0x7f962c003520 con 0x7f9630002860 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96300065c0 con 0x7f9630002860 2024-09-13T13:52:58.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9644003e10 con 0x7f9630002860 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 10 monclient: dump: 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.749 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2089629857 0 0) 0x7f96300065c0 con 0x7f9630002860 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f963000a160 con 0x7f9630002860 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f96440041b0 con 0x7f9630002860 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:58.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9630002860 legacy=0x7f9630004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.724+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9630002860 legacy=0x7f9630004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.730+0000 7f964ffda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:58.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.730+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9630009e70 con 0x7f9630009130 2024-09-13T13:52:58.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f964dd76640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9630009130 legacy=0x7f96300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899224646 0 0) 0x7f9630009e70 con 0x7f9630009130 2024-09-13T13:52:58.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient(hunting): my global_id is 4539 2024-09-13T13:52:58.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96300065c0 con 0x7f9630009130 2024-09-13T13:52:58.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 378223294 0 0) 0x7f96300065c0 con 0x7f9630009130 2024-09-13T13:52:58.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9630009e70 con 0x7f9630009130 2024-09-13T13:52:58.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f9644005b00 con 0x7f9630009130 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 10 monclient: dump: 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.755 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1848925197 0 0) 0x7f9630009e70 con 0x7f9630009130 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.733+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 --> v1:172.21.15.153:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f963000c410 con 0x7f9630009130 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 <== mon.3 v1:172.21.15.153:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7559 (unknown 0 0 3162996360) 0x7f9644006f50 con 0x7f9630009130 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f963e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:58.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f963e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9630009130 legacy=0x7f96300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f963e7fc640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6790/0 conn(0x7f9630009130 legacy=0x7f96300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:58.757 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 102358, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-13T13:52:58.758 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.759 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.760 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:58.761 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.762 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:58.763 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.765 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.766 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.767 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.768 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:58.769 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.770 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:58.773 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.774 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.775 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:58.776 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.777 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.778 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:58.778 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:58.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.153:6812/1206688214 conn(0x7f962c0406d0 legacy=0x7f962c042b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 10 monclient: shutdown 2024-09-13T13:52:58.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 >> v1:172.21.15.121:6790/0 conn(0x7f9648138fb0 legacy=0x7f96481d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964e577640 1 -- 172.21.15.121:0/2780657991 reap_dead start 2024-09-13T13:52:58.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964e577640 1 -- 172.21.15.121:0/2780657991 reap_dead start 2024-09-13T13:52:58.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 shutdown_connections 2024-09-13T13:52:58.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 >> 172.21.15.121:0/2780657991 conn(0x7f96480ab800 msgr2=0x7f964812c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 shutdown_connections 2024-09-13T13:52:58.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.734+0000 7f964ffda640 1 -- 172.21.15.121:0/2780657991 wait complete. 2024-09-13T13:52:58.780 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:58.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-13T13:52:58.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-13T13:52:58.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.877+0000 7f604facd640 10 monclient: get_monmap_and_config 2024-09-13T13:52:58.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.877+0000 7f604facd640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.877+0000 7f604facd640 10 monclient: monmap: 2024-09-13T13:52:58.879 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.879 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.879 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:58.879073+0000 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:58.879073+0000 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.880 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.881 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.881 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.881 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.881 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.877+0000 7f604facd640 1 Processor -- start 2024-09-13T13:52:58.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 1 -- start start 2024-09-13T13:52:58.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient: init 2024-09-13T13:52:58.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient: _add_conns ranks=[2,4,5,8,0,7,1,6,3] 2024-09-13T13:52:58.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient(hunting): picked mon.noname-b con 0x7f6048138bd0 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient(hunting): picked mon.noname-c con 0x7f6048138fb0 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient(hunting): picked mon.noname-h con 0x7f6048139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604809e100 con 0x7f6048138bd0 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604809db40 con 0x7f6048138fb0 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604809de20 con 0x7f6048139420 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604facd640 10 monclient(hunting): authenticate will time out at 858.508740s 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604d869640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f6048138bd0 0x7f60480a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55046/0 (socket says 172.21.15.121:55046) 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.878+0000 7f604d869640 1 -- 172.21.15.121:0/1786770791 learned_addr learned my addr 172.21.15.121:0/1786770791 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.880+0000 7f604d869640 1 -- 172.21.15.121:0/1786770791 >> v1:172.21.15.121:6790/0 conn(0x7f6048138bd0 legacy=0x7f60480a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604d068640 1 -- 172.21.15.121:0/1786770791 >> v1:172.21.15.121:6791/0 conn(0x7f6048138fb0 legacy=0x7f604813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16778937 0 0) 0x7f604809e100 con 0x7f6048138bd0 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): my global_id is 4592 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6034003520 con 0x7f6048138bd0 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4078774943 0 0) 0x7f604809db40 con 0x7f6048138fb0 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient(hunting): my global_id is 4558 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f604809e100 con 0x7f6048138fb0 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604e06a640 1 -- 172.21.15.121:0/1786770791 >> v1:172.21.15.153:6791/0 conn(0x7f6048139420 legacy=0x7f6048141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 10915929 0 0) 0x7f6034003520 con 0x7f6048138bd0 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 >> v1:172.21.15.153:6791/0 conn(0x7f6048139420 legacy=0x7f6048141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 >> v1:172.21.15.121:6791/0 conn(0x7f6048138fb0 legacy=0x7f604813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 monclient: found mon.noname-b 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60480a5da0 con 0x7f6048138bd0 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6044001ae0 con 0x7f6048138bd0 2024-09-13T13:52:58.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 5 monclient: authenticate success, global_id 4592 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: dump: 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.890 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.892 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:58.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f60440048a0 con 0x7f6048138bd0 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 1 -- 172.21.15.121:0/1786770791 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6044004e10 con 0x7f6048138bd0 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 4 set_mon_vals no callback set 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604c867640 10 monclient: dump: 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.894 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 10 monclient: shutdown 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.881+0000 7f604facd640 1 -- 172.21.15.121:0/1786770791 >> v1:172.21.15.121:6790/0 conn(0x7f6048138bd0 legacy=0x7f60480a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 1 -- 172.21.15.121:0/1786770791 shutdown_connections 2024-09-13T13:52:58.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 1 -- 172.21.15.121:0/1786770791 >> 172.21.15.121:0/1786770791 conn(0x7f60480ab800 msgr2=0x7f60480abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 1 -- 172.21.15.121:0/1786770791 shutdown_connections 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 1 -- 172.21.15.121:0/1786770791 wait complete. 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 10 monclient: build_initial_monmap 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 10 monclient: monmap: 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:58.896 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:58.897 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:58.897 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.882+0000 7f604facd640 1 Processor -- start 2024-09-13T13:52:58.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 1 -- start start 2024-09-13T13:52:58.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient: init 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient: _add_conns ranks=[0,4,2,3,6,7,8,5,1] 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient(hunting): picked mon.noname-a con 0x7f6048138bd0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient(hunting): picked mon.noname-c con 0x7f6048138fb0 addr v1:172.21.15.121:6791/0 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient(hunting): picked mon.noname-b con 0x7f6048139420 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6048130610 con 0x7f6048138bd0 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60480a76a0 con 0x7f6048139420 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60480a59f0 con 0x7f6048138fb0 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604facd640 10 monclient(hunting): authenticate will time out at 858.513471s 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604e06a640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f6048139420 0x7f60481d5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55062/0 (socket says 172.21.15.121:55062) 2024-09-13T13:52:58.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604d068640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f6048138fb0 0x7f604812f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33606/0 (socket says 172.21.15.121:33606) 2024-09-13T13:52:58.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.883+0000 7f604e06a640 1 -- 172.21.15.121:0/4102919628 learned_addr learned my addr 172.21.15.121:0/4102919628 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:58.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f604d869640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.121:6789/0 conn(0x7f6048138bd0 legacy=0x7f604812eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f604d068640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.121:6791/0 conn(0x7f6048138fb0 legacy=0x7f604812f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f604e06a640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.121:6790/0 conn(0x7f6048139420 legacy=0x7f60481d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200692091 0 0) 0x7f6048130610 con 0x7f6048138bd0 2024-09-13T13:52:58.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): my global_id is 4653 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f602c003520 con 0x7f6048138bd0 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3093978781 0 0) 0x7f60480a76a0 con 0x7f6048139420 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): my global_id is 4601 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6048130610 con 0x7f6048139420 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348780817 0 0) 0x7f60480a59f0 con 0x7f6048138fb0 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 10 monclient(hunting): my global_id is 4567 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.885+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60480a76a0 con 0x7f6048138fb0 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2512058830 0 0) 0x7f602c003520 con 0x7f6048138bd0 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60480a59f0 con 0x7f6048138bd0 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6044004d90 con 0x7f6048138bd0 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.909 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3238235119 0 0) 0x7f6048130610 con 0x7f6048139420 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f602c003520 con 0x7f6048139420 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2496019299 0 0) 0x7f60480a76a0 con 0x7f6048138fb0 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6048130610 con 0x7f6048138fb0 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6040002090 con 0x7f6048139420 2024-09-13T13:52:58.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6038003610 con 0x7f6048138fb0 2024-09-13T13:52:58.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient(hunting): dump: 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.913 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 923208253 0 0) 0x7f60480a59f0 con 0x7f6048138bd0 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.121:6791/0 conn(0x7f6048138fb0 legacy=0x7f604812f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.121:6790/0 conn(0x7f6048139420 legacy=0x7f60481d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 monclient: found mon.a 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:58.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6048132970 con 0x7f6048138bd0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: _finish_auth 0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 5 monclient: authenticate success, global_id 4653 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 10 monclient: _renew_subs 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f60481320d0 con 0x7f6048138bd0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 10 monclient: _renew_subs 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6048131dc0 con 0x7f6048138bd0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6044001d30 con 0x7f6048138bd0 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:58.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f60440048a0 con 0x7f6048138bd0 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603ffff640 4 set_mon_vals no callback set 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.886+0000 7f603e7fc640 10 monclient: dump: 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.916 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.887+0000 7f604facd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.887+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f60300065c0 con 0x7f6030002860 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.887+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f6044011c20 con 0x7f6048138bd0 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.887+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f604404e0b0 con 0x7f6048138bd0 2024-09-13T13:52:58.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.887+0000 7f604e06a640 10 monclient: get_auth_request con 0x7f602c040710 auth_method 0 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.889+0000 7f604d068640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6791/0 conn(0x7f6030002860 legacy=0x7f6030004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754753885 0 0) 0x7f60300065c0 con 0x7f6030002860 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient(hunting): my global_id is 4568 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60480a59f0 con 0x7f6030002860 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1179365124 0 0) 0x7f60480a59f0 con 0x7f6030002860 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f60300065c0 con 0x7f6030002860 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6038003d60 con 0x7f6030002860 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient: dump: 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.919 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 198474117 0 0) 0x7f60300065c0 con 0x7f6030002860 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f603000a160 con 0x7f6030002860 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6038004050 con 0x7f6030002860 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6791/0 conn(0x7f6030002860 legacy=0x7f6030004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.890+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6791/0 conn(0x7f6030002860 legacy=0x7f6030004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.896+0000 7f604facd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.896+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6030009e70 con 0x7f6030009130 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f604d068640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6791/0 conn(0x7f6030009130 legacy=0x7f60300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 769153032 0 0) 0x7f6030009e70 con 0x7f6030009130 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f603e7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:52:58.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f603e7fc640 10 monclient(hunting): my global_id is 4577 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.899+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60300065c0 con 0x7f6030009130 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1485970431 0 0) 0x7f60300065c0 con 0x7f6030009130 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6030009e70 con 0x7f6030009130 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6038005b00 con 0x7f6030009130 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 10 monclient: dump: 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:58.921 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:58.922 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 627261181 0 0) 0x7f6030009e70 con 0x7f6030009130 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 --> v1:172.21.15.153:6791/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f603000c410 con 0x7f6030009130 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 <== mon.5 v1:172.21.15.153:6791/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7726 (unknown 0 0 2435109180) 0x7f6038007e40 con 0x7f6030009130 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6791/0 conn(0x7f6030009130 legacy=0x7f60300095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f603e7fc640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6791/0 conn(0x7f6030009130 legacy=0x7f60300095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.923 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 102533, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.924 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.925 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.926 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.927 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:58.928 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:58.929 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.930 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.931 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.932 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.933 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.934 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.935 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:58.937 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.938 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:58.939 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.940 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.941 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "osd": [ 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:58.942 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 2 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:58.943 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:58.944 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:58.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.153:6812/1206688214 conn(0x7f602c040710 legacy=0x7f602c042bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f604facd640 10 monclient: shutdown 2024-09-13T13:52:58.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f604facd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:58.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 >> v1:172.21.15.121:6789/0 conn(0x7f6048138bd0 legacy=0x7f604812eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:58.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f604e06a640 1 -- 172.21.15.121:0/4102919628 reap_dead start 2024-09-13T13:52:58.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.900+0000 7f604e06a640 1 -- 172.21.15.121:0/4102919628 reap_dead start 2024-09-13T13:52:58.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.901+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 shutdown_connections 2024-09-13T13:52:58.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.901+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 >> 172.21.15.121:0/4102919628 conn(0x7f60480ab800 msgr2=0x7f604812c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:58.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.901+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 shutdown_connections 2024-09-13T13:52:58.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:58.901+0000 7f604facd640 1 -- 172.21.15.121:0/4102919628 wait complete. 2024-09-13T13:52:58.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:58.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-13T13:52:58.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-13T13:52:59.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.042+0000 7fda46860640 10 monclient: get_monmap_and_config 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.042+0000 7fda46860640 10 monclient: build_initial_monmap 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.042+0000 7fda46860640 10 monclient: monmap: 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:59.044429+0000 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:59.044429+0000 2024-09-13T13:52:59.045 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:59.046 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:59.047 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 1 Processor -- start 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 1 -- start start 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient: init 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient: _add_conns ranks=[1,0,5,7,3,4,6,2,8] 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient(hunting): picked mon.noname-f con 0x7fda40138bd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient(hunting): picked mon.noname-a con 0x7fda40138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient(hunting): picked mon.noname-h con 0x7fda40139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4009e100 con 0x7fda40138fb0 2024-09-13T13:52:59.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4009db40 con 0x7fda40138bd0 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4009de20 con 0x7fda40139420 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.043+0000 7fda46860640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.044+0000 7fda46860640 10 monclient(hunting): authenticate will time out at 858.674008s 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.044+0000 7fda3f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fda40138fb0 0x7fda4013df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59084/0 (socket says 172.21.15.121:59084) 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.044+0000 7fda3f7fe640 1 -- 172.21.15.121:0/3200154422 learned_addr learned my addr 172.21.15.121:0/3200154422 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3f7fe640 1 -- 172.21.15.121:0/3200154422 >> v1:172.21.15.121:6789/0 conn(0x7fda40138fb0 legacy=0x7fda4013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336755777 0 0) 0x7fda4009e100 con 0x7fda40138fb0 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda44dfd640 1 -- 172.21.15.121:0/3200154422 >> v1:172.21.15.153:6791/0 conn(0x7fda40139420 legacy=0x7fda40141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): my global_id is 4662 2024-09-13T13:52:59.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda2c003520 con 0x7fda40138fb0 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3ffff640 1 -- 172.21.15.121:0/3200154422 >> v1:172.21.15.153:6789/0 conn(0x7fda40138bd0 legacy=0x7fda400a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 626376309 0 0) 0x7fda4009de20 con 0x7fda40139420 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient(hunting): my global_id is 4586 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda4009e100 con 0x7fda40139420 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3331385504 0 0) 0x7fda2c003520 con 0x7fda40138fb0 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 >> v1:172.21.15.153:6791/0 conn(0x7fda40139420 legacy=0x7fda40141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 >> v1:172.21.15.153:6789/0 conn(0x7fda40138bd0 legacy=0x7fda400a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: _finish_hunting 0 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 monclient: found mon.noname-a 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:59.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda400a5da0 con 0x7fda40138fb0 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: _finish_auth 0 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: _check_auth_tickets 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fda30001ae0 con 0x7fda40138fb0 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: dump: 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.058 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda46860640 5 monclient: authenticate success, global_id 4662 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda46860640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda300048a0 con 0x7fda40138fb0 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:59.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 1 -- 172.21.15.121:0/3200154422 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fda30004e10 con 0x7fda40138fb0 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda46860640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda46860640 4 set_mon_vals no callback set 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.046+0000 7fda3effd640 10 monclient: dump: 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.060 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 10 monclient: shutdown 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 1 -- 172.21.15.121:0/3200154422 >> v1:172.21.15.121:6789/0 conn(0x7fda40138fb0 legacy=0x7fda4013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 1 -- 172.21.15.121:0/3200154422 shutdown_connections 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 1 -- 172.21.15.121:0/3200154422 >> 172.21.15.121:0/3200154422 conn(0x7fda400ab800 msgr2=0x7fda400abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 1 -- 172.21.15.121:0/3200154422 shutdown_connections 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 1 -- 172.21.15.121:0/3200154422 wait complete. 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 10 monclient: build_initial_monmap 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.047+0000 7fda46860640 10 monclient: monmap: 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:59.062 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 1 Processor -- start 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 1 -- start start 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 10 monclient: init 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 10 monclient: _add_conns ranks=[1,0,7,3,6,4,8,2,5] 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 10 monclient(hunting): picked mon.noname-f con 0x7fda40138bd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 10 monclient(hunting): picked mon.noname-a con 0x7fda40138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.048+0000 7fda46860640 10 monclient(hunting): picked mon.noname-i con 0x7fda40139420 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda46860640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda400a7bb0 con 0x7fda40138fb0 2024-09-13T13:52:59.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda46860640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda400a59b0 con 0x7fda40138bd0 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda46860640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda40123bc0 con 0x7fda40139420 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda46860640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda46860640 10 monclient(hunting): authenticate will time out at 858.679049s 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda3f7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fda40138fb0 0x7fda40135360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59098/0 (socket says 172.21.15.121:59098) 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.049+0000 7fda3f7fe640 1 -- 172.21.15.121:0/1808942224 learned_addr learned my addr 172.21.15.121:0/1808942224 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3f7fe640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.121:6789/0 conn(0x7fda40138fb0 legacy=0x7fda40135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4172470900 0 0) 0x7fda400a5da0 con 0x7fda40138fb0 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): my global_id is 4671 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda20003520 con 0x7fda40138fb0 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3ffff640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6789/0 conn(0x7fda40138bd0 legacy=0x7fda40134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda44dfd640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda40139420 legacy=0x7fda401d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2989900972 0 0) 0x7fda20003520 con 0x7fda40138fb0 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda400a5da0 con 0x7fda40138fb0 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fda30004d90 con 0x7fda40138fb0 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): dump: 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.065 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 904790933 0 0) 0x7fda40123bc0 con 0x7fda40139420 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient(hunting): my global_id is 104146 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda20003520 con 0x7fda40139420 2024-09-13T13:52:59.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 955460420 0 0) 0x7fda400a5da0 con 0x7fda40138fb0 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda40139420 legacy=0x7fda401d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6789/0 conn(0x7fda40138bd0 legacy=0x7fda40134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: _finish_hunting 0 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 monclient: found mon.a 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda400a7680 con 0x7fda40138fb0 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: _finish_auth 0 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: _check_auth_tickets 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 5 monclient: authenticate success, global_id 4671 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 10 monclient: _renew_subs 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:59.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fda401238e0 con 0x7fda40138fb0 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 10 monclient: _renew_subs 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fda40135a70 con 0x7fda40138fb0 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fda30001d30 con 0x7fda40138fb0 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fda30001a80 con 0x7fda40138fb0 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3e7fc640 4 set_mon_vals no callback set 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.051+0000 7fda3cff9640 10 monclient: dump: 2024-09-13T13:52:59.068 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.069 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.052+0000 7fda46860640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.052+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fda240065c0 con 0x7fda24002860 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.052+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fda30011c20 con 0x7fda40138fb0 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.052+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fda3004e260 con 0x7fda40138fb0 2024-09-13T13:52:59.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.053+0000 7fda44dfd640 10 monclient: get_auth_request con 0x7fda2003fa20 auth_method 0 2024-09-13T13:52:59.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.054+0000 7fda3ffff640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda24002860 legacy=0x7fda24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551024466 0 0) 0x7fda240065c0 con 0x7fda24002860 2024-09-13T13:52:59.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient(hunting): my global_id is 104155 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda400a5da0 con 0x7fda24002860 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3676305281 0 0) 0x7fda400a5da0 con 0x7fda24002860 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda240065c0 con 0x7fda24002860 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fda28003610 con 0x7fda24002860 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient: dump: 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3261939703 0 0) 0x7fda240065c0 con 0x7fda24002860 2024-09-13T13:52:59.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda2400a160 con 0x7fda24002860 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fda280057c0 con 0x7fda24002860 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.055+0000 7fda3cff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.056+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda24002860 legacy=0x7fda24004d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.056+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda24002860 legacy=0x7fda24004d00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.062+0000 7fda46860640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.062+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fda24009e70 con 0x7fda24009130 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3ffff640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda24009130 legacy=0x7fda240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50205920 0 0) 0x7fda24009e70 con 0x7fda24009130 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient(hunting): my global_id is 104164 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda240065c0 con 0x7fda24009130 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3240299656 0 0) 0x7fda240065c0 con 0x7fda24009130 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fda24009e70 con 0x7fda24009130 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fda28003ed0 con 0x7fda24009130 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 10 monclient: dump: 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2330900965 0 0) 0x7fda24009e70 con 0x7fda24009130 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.064+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 --> v1:172.21.15.153:6792/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda2400c410 con 0x7fda24009130 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 <== mon.7 v1:172.21.15.153:6792/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+7558 (unknown 0 0 1760427880) 0x7fda28004170 con 0x7fda24009130 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda3cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda3cff9640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda24009130 legacy=0x7fda240095c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda3cff9640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6792/0 conn(0x7fda24009130 legacy=0x7fda240095c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "peon", 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: "election_epoch": 28, 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 22341, 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum": [ 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: 0, 2024-09-13T13:52:59.079 INFO:teuthology.orchestra.run.smithi121.stdout: 1, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 2, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 3, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 4, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 5, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 6, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 7, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: 8 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_age": 2, 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:59.080 INFO:teuthology.orchestra.run.smithi121.stdout: "required_con": "2450521708007866372", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "required_mon": [ 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:59.081 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_con": "4540719139924082687", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "quorum_mon": [ 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:59.082 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "outside_quorum": [], 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "extra_probe_peers": [], 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "sync_provider": [], 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "monmap": { 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 1, 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "fsid": "21ebbdf6-8985-4682-9234-cdd2d2c1c2db", 2024-09-13T13:52:59.083 INFO:teuthology.orchestra.run.smithi121.stdout: "modified": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "created": "2024-09-13T13:50:47.391384Z", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release": 19, 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "min_mon_release_name": "squid", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "election_strategy": 3, 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "disallowed_leaders: ": "", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false, 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "tiebreaker_mon": "", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "removed_ranks: ": "", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "features": { 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "persistent": [ 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "kraken", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "luminous", 2024-09-13T13:52:59.084 INFO:teuthology.orchestra.run.smithi121.stdout: "mimic", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap-prune", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "nautilus", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "elector-pinging", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "reef", 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "squid" 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "optional": [] 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "mons": [ 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 0, 2024-09-13T13:52:59.085 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "a", 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789", 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6789/0", 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6789/0", 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 1, 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "f", 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789", 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6789/0", 2024-09-13T13:52:59.087 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 2, 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "b", 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790", 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.088 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6790/0", 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6790/0", 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 3, 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "g", 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.089 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790", 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6790/0", 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.090 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 4, 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "c", 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791", 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.091 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6791/0", 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6791/0", 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 5, 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "h", 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791", 2024-09-13T13:52:59.092 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6791/0", 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6791/0", 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 6, 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "d", 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.093 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792", 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6792/0", 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6792/0", 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 7, 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "i", 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792", 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.153:6792/0", 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.153:6792/0", 2024-09-13T13:52:59.095 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "rank": 8, 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "e", 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addrs": { 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793", 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 0 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.096 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6793/0", 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "public_addr": "172.21.15.121:6793/0", 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "priority": 0, 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "weight": 0, 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_location": "{}" 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "feature_map": { 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "mon": [ 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:59.097 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: "client": [ 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: "features": "0x3f03dffffffdffff", 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: "release": "squid", 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 1 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout: "stretch_mode": false 2024-09-13T13:52:59.098 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:52:59.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.153:6812/1206688214 conn(0x7fda2003fa20 legacy=0x7fda20041ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 10 monclient: shutdown 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 >> v1:172.21.15.121:6789/0 conn(0x7fda40138fb0 legacy=0x7fda40135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda44dfd640 1 -- 172.21.15.121:0/1808942224 reap_dead start 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda44dfd640 1 -- 172.21.15.121:0/1808942224 reap_dead start 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 shutdown_connections 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 >> 172.21.15.121:0/1808942224 conn(0x7fda400ab800 msgr2=0x7fda4012c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 shutdown_connections 2024-09-13T13:52:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.065+0000 7fda46860640 1 -- 172.21.15.121:0/1808942224 wait complete. 2024-09-13T13:52:59.101 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-13T13:52:59.101 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-13T13:52:59.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-13T13:52:59.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.197+0000 7fccd56d4640 10 monclient: get_monmap_and_config 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.197+0000 7fccd56d4640 10 monclient: build_initial_monmap 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.197+0000 7fccd56d4640 10 monclient: monmap: 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:52:59.198763+0000 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:52:59.198763+0000 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:59.199 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:59.200 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:59.200 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:59.200 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:59.200 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:59.200 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:59.200 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.197+0000 7fccd56d4640 1 Processor -- start 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.197+0000 7fccd56d4640 1 -- start start 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.197+0000 7fccd56d4640 10 monclient: init 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient: _add_conns ranks=[5,7,6,4,8,2,3,0,1] 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient(hunting): picked mon.noname-h con 0x7fccd0138c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient(hunting): picked mon.noname-i con 0x7fccd013bfc0 addr v1:172.21.15.153:6792/0 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient(hunting): picked mon.noname-d con 0x7fccd013fad0 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd00a5e80 con 0x7fccd0138c50 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd0139a60 con 0x7fccd013fad0 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd009df00 con 0x7fccd013bfc0 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fccd56d4640 10 monclient(hunting): authenticate will time out at 858.828141s 2024-09-13T13:52:59.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fcccf7fe640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fccd013fad0 0x7fccd0141ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51448/0 (socket says 172.21.15.121:51448) 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.198+0000 7fcccf7fe640 1 -- 172.21.15.121:0/504482295 learned_addr learned my addr 172.21.15.121:0/504482295 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccf7fe640 1 -- 172.21.15.121:0/504482295 >> v1:172.21.15.121:6792/0 conn(0x7fccd013fad0 legacy=0x7fccd0141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fccce7fc640 1 -- 172.21.15.121:0/504482295 >> v1:172.21.15.153:6792/0 conn(0x7fccd013bfc0 legacy=0x7fccd013e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fccceffd640 1 -- 172.21.15.121:0/504482295 >> v1:172.21.15.153:6791/0 conn(0x7fccd0138c50 legacy=0x7fccd00a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147631370 0 0) 0x7fccd0139a60 con 0x7fccd013fad0 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): my global_id is 4650 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccb8003520 con 0x7fccd013fad0 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3746935142 0 0) 0x7fccd009df00 con 0x7fccd013bfc0 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): my global_id is 104173 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd0139a60 con 0x7fccd013bfc0 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1199655965 0 0) 0x7fccc80014a0 con 0x7fccd0138c50 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient(hunting): my global_id is 4595 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd009df00 con 0x7fccd0138c50 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2188864704 0 0) 0x7fccb8003520 con 0x7fccd013fad0 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 >> v1:172.21.15.153:6792/0 conn(0x7fccd013bfc0 legacy=0x7fccd013e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 >> v1:172.21.15.153:6791/0 conn(0x7fccd0138c50 legacy=0x7fccd00a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: _finish_hunting 0 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 monclient: found mon.noname-d 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd009d940 con 0x7fccd013fad0 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: _finish_auth 0 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: _check_auth_tickets 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fccd56d4640 5 monclient: authenticate success, global_id 4650 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fccc0001ae0 con 0x7fccd013fad0 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:52:59.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: dump: 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fccd56d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fccc00048a0 con 0x7fccd013fad0 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fccd56d4640 10 monclient: get_monmap_and_config success 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fccd56d4640 4 set_mon_vals no callback set 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 1 -- 172.21.15.121:0/504482295 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fccc0004e50 con 0x7fccd013fad0 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.200+0000 7fcccdffb640 10 monclient: dump: 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.217 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 10 monclient: shutdown 2024-09-13T13:52:59.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 1 -- 172.21.15.121:0/504482295 >> v1:172.21.15.121:6792/0 conn(0x7fccd013fad0 legacy=0x7fccd0141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 1 -- 172.21.15.121:0/504482295 shutdown_connections 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 1 -- 172.21.15.121:0/504482295 >> 172.21.15.121:0/504482295 conn(0x7fccd0069360 msgr2=0x7fccd00ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 1 -- 172.21.15.121:0/504482295 shutdown_connections 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 1 -- 172.21.15.121:0/504482295 wait complete. 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.201+0000 7fccd56d4640 10 monclient: build_initial_monmap 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 10 monclient: monmap: 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:52:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 1 Processor -- start 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 1 -- start start 2024-09-13T13:52:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 10 monclient: init 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 10 monclient: _reopen_session rank -1 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 10 monclient: _add_conns ranks=[5,6,2,4,7,3,0,1,8] 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 10 monclient(hunting): picked mon.noname-h con 0x7fccd0138c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.202+0000 7fccd56d4640 10 monclient(hunting): picked mon.noname-d con 0x7fccd013bfc0 addr v1:172.21.15.121:6792/0 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccd56d4640 10 monclient(hunting): picked mon.noname-b con 0x7fccd013fad0 addr v1:172.21.15.121:6790/0 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccd56d4640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd0136140 con 0x7fccd013fad0 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccd56d4640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd00a8fe0 con 0x7fccd0138c50 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccd56d4640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd00a72f0 con 0x7fccd013bfc0 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccd56d4640 10 monclient(hunting): _renew_subs 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccd56d4640 10 monclient(hunting): authenticate will time out at 858.833046s 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccce7fc640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fccd013bfc0 0x7fccd01d3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51452/0 (socket says 172.21.15.121:51452) 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.203+0000 7fccce7fc640 1 -- 172.21.15.121:0/3315616309 learned_addr learned my addr 172.21.15.121:0/3315616309 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccce7fc640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.121:6792/0 conn(0x7fccd013bfc0 legacy=0x7fccd01d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fcccf7fe640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.121:6790/0 conn(0x7fccd013fad0 legacy=0x7fccd01d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccceffd640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.153:6791/0 conn(0x7fccd0138c50 legacy=0x7fccd01ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1248025369 0 0) 0x7fccd00a72f0 con 0x7fccd013bfc0 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): my global_id is 4659 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccbc003520 con 0x7fccd013bfc0 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682343475 0 0) 0x7fccd0136140 con 0x7fccd013fad0 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): my global_id is 4610 2024-09-13T13:52:59.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd00a72f0 con 0x7fccd013fad0 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 179995543 0 0) 0x7fccd00a8fe0 con 0x7fccd0138c50 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _negotiate 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): my global_id is 4604 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fccd0136140 con 0x7fccd0138c50 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3444859002 0 0) 0x7fccd00a72f0 con 0x7fccd013fad0 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccd00a8fe0 con 0x7fccd013fad0 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fccc0001a60 con 0x7fccd013fad0 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): dump: 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.223 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:52:59.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1074281853 0 0) 0x7fccbc003520 con 0x7fccd013bfc0 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccd00a72f0 con 0x7fccd013bfc0 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1336459372 0 0) 0x7fccd0136140 con 0x7fccd0138c50 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fccbc003520 con 0x7fccd0138c50 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fccc4003990 con 0x7fccd013bfc0 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): dump: 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.225 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fccc8001440 con 0x7fccd0138c50 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:52:59.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient(hunting): dump: 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3997804251 0 0) 0x7fccd00a8fe0 con 0x7fccd013fad0 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.121:6792/0 conn(0x7fccd013bfc0 legacy=0x7fccd01d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.153:6791/0 conn(0x7fccd0138c50 legacy=0x7fccd01ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient: _finish_hunting 0 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 monclient: found mon.b 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd0135970 con 0x7fccd013fad0 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient: _finish_auth 0 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 10 monclient: _check_auth_tickets 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:52:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 5 monclient: authenticate success, global_id 4610 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 10 monclient: _renew_subs 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fccd00a77e0 con 0x7fccd013fad0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 10 monclient: _renew_subs 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.205+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fccd01cea70 con 0x7fccd013fad0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fccc0001d30 con 0x7fccd013fad0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fccc0001580 con 0x7fccd013fad0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fcccd7fa640 4 set_mon_vals no callback set 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 10 monclient: dump: 2024-09-13T13:52:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:52:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.206+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fccc0011b60 con 0x7fccd013fad0 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.207+0000 7fccd56d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.207+0000 7fccd56d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.207+0000 7fccd56d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.207+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb0005080 con 0x7fccd013fad0 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.207+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fccc004e290 con 0x7fccd013fad0 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.207+0000 7fccceffd640 10 monclient: get_auth_request con 0x7fccbc040540 auth_method 0 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.211+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fccc0017900 con 0x7fccd013fad0 2024-09-13T13:52:59.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.211+0000 7fccb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:52:59.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.211+0000 7fccb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:52:59.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.461+0000 7fccd56d4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-13T13:52:59.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.461+0000 7fccd56d4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-13T13:52:59.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.461+0000 7fccd56d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:52:59.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.461+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fccb0005b80 con 0x7fccd013fad0 2024-09-13T13:52:59.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fccb77fe640 1 -- 172.21.15.121:0/3315616309 <== mon.2 v1:172.21.15.121:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fccc0017200 con 0x7fccd013fad0 2024-09-13T13:52:59.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fccb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-13T13:52:59.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fccb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=9,logm=60} crc {auth=1480034411,config=3541143443,health=2753555641,kv=585613442,logm=3725186819}) 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=1480034411,config=3541143443,health=2374691465,kv=3378697076,logm=1203863199}) 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.467 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=2230051351,config=3541143443,health=3897446751,kv=3378697076,logm=421856334}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=17,config=2,health=11,kv=11,logm=59} crc {auth=1373085099,config=3541143443,health=344943315,kv=3378697076,logm=4177016081}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1278050900,config=3541143443,health=3897446751,kv=1178025407,logm=3800350253}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=17,config=2,health=11,kv=11,logm=59} crc {auth=3266744253,config=3541143443,health=588522927,kv=3378697076,logm=2271183575}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=16,config=2,health=12,kv=11,logm=59} crc {auth=895479617,config=3541143443,health=3897446751,kv=3378697076,logm=591757733}) 2024-09-13T13:52:59.468 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.469 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=1658641329,config=3541143443,health=3897446751,kv=2258573378,logm=656412735}) 2024-09-13T13:52:59.469 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.463+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=1480034411,config=3541143443,health=3897446751,kv=3378697076,logm=3863497293}) 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.465+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.153:6812/1206688214 conn(0x7fccbc040540 legacy=0x7fccbc042a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.465+0000 7fccd56d4640 10 monclient: shutdown 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.465+0000 7fccd56d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.465+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 >> v1:172.21.15.121:6790/0 conn(0x7fccd013fad0 legacy=0x7fccd01d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.465+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 shutdown_connections 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.465+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 >> 172.21.15.121:0/3315616309 conn(0x7fccd0069360 msgr2=0x7fccd0136800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.466+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 shutdown_connections 2024-09-13T13:52:59.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:52:59.466+0000 7fccd56d4640 1 -- 172.21.15.121:0/3315616309 wait complete. 2024-09-13T13:52:59.482 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.472+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.472+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094333027}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481063864}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482114495}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=737457886}) 2024-09-13T13:52:59.483 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3171420051}) 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1268287868}) 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1226330856}) 2024-09-13T13:52:59.484 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2074681486}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.473+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=135539595}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086312259}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409426065}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.485 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321754292}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=727490610}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=669213695}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2559850394}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.486 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3856411790}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3445915495}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.474+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3340228531}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801936190}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=330681036}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.475+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644300590}) 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.487 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776959486}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1580459407}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3185295472}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.488 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2162261886}) 2024-09-13T13:52:59.489 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.489 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1543013940}) 2024-09-13T13:52:59.489 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.476+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2213339107}) 2024-09-13T13:52:59.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-13T13:52:59.495 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.495 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.495 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=813814397}) 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3229424878}) 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4211378004}) 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.496 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1308461709}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2574283697}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=534288246}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2085722518}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2567679988}) 2024-09-13T13:52:59.497 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.477+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=69151646}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391306480}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3772040923}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316038893}) 2024-09-13T13:52:59.498 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3048403806}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3911999502}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1060929016}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=51186704}) 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.499 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2329948942}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.478+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3770564994}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=28} crc {logm=1343005224,mdsmap=738663024,mgr=1592222567,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1615466601,mon_config_key=2364086306,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1847723895,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=478715586}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=30} crc {logm=2336761153,mdsmap=738663024,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=2935912735,mon_config_key=2038595929,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1847723895,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1916646463}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2,osdmap=30} crc {logm=242931178,mdsmap=738663024,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=3241563255,mon_config_key=1776337002,monmap=4128118807,nvmeofgw=598337686,osd_metadata=4231186811,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=4203030188}) 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.500 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.479+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=19,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=2910180788,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1615466601,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1847723895,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=2659687143}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=29} crc {logm=2590164382,mdsmap=738663024,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1615466601,mon_config_key=1060064773,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1847723895,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=748466961}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=17,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=34} crc {logm=3290943635,mdsmap=738663024,mgr=4034175311,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=377043642,mon_config_key=2364086306,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1847723895,osd_pg_creating=4187763148,osd_snap=4133920409,osdmap=1787773392}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=1,osdmap=31} crc {logm=2336761153,mdsmap=738663024,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1615466601,mon_config_key=695915725,monmap=4128118807,nvmeofgw=598337686,osd_metadata=172732233,osd_pg_creating=4187763147,osd_snap=3823785925,osdmap=583145139}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=16,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=2,osdmap=35} crc {logm=2167692719,mdsmap=3106100972,mgr=2253636195,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1953283007,mon_config_key=3529659646,monmap=4128118807,nvmeofgw=598337686,osd_metadata=1881093266,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1156823609}) 2024-09-13T13:52:59.501 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.480+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=31} crc {logm=2336761153,mdsmap=738663024,mgr=258987911,mgr_command_descs=3192186153,mgr_metadata=1781127783,mgrstat=1615466601,mon_config_key=1776337002,monmap=4201785531,nvmeofgw=598337686,osd_metadata=1847723895,osd_pg_creating=4187763147,osd_snap=4133920409,osdmap=1570278723}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=4055431954}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=3787569544}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=4055431954}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=3091252583}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=4055431954}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=21} crc {osdmap=947086095}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=4055431954}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=20} crc {osdmap=3989437775}) 2024-09-13T13:52:59.502 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.503 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=4055431954}) 2024-09-13T13:52:59.503 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=20} crc {osdmap=2956576242}) 2024-09-13T13:52:59.503 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-13T13:52:59.503 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=4055431954}) 2024-09-13T13:52:59.503 INFO:tasks.ceph.mon.d.smithi121.stderr:2024-09-13T13:52:59.481+0000 7fb7d659b640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=407562711}) 2024-09-13T13:53:00.490 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-13T13:53:00.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.596+0000 7f12db5ab640 10 monclient: get_monmap_and_config 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.596+0000 7f12db5ab640 10 monclient: build_initial_monmap 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.596+0000 7f12db5ab640 10 monclient: monmap: 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:00.597996+0000 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:00.597996+0000 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:00.598 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:00.599 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.596+0000 7f12db5ab640 1 Processor -- start 2024-09-13T13:53:00.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 1 -- start start 2024-09-13T13:53:00.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient: init 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient: _add_conns ranks=[1,4,7,3,0,8,5,2,6] 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient(hunting): picked mon.noname-f con 0x7f12d4138c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient(hunting): picked mon.noname-c con 0x7f12d413bfc0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient(hunting): picked mon.noname-i con 0x7f12d413fad0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d40a5e80 con 0x7f12d4138c50 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d4139a60 con 0x7f12d413bfc0 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d409df00 con 0x7f12d413fad0 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12db5ab640 10 monclient(hunting): authenticate will time out at 860.227798s 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12d8b46640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f12d413bfc0 0x7f12d413e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33620/0 (socket says 172.21.15.121:33620) 2024-09-13T13:53:00.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.597+0000 7f12d8b46640 1 -- 172.21.15.121:0/71540438 learned_addr learned my addr 172.21.15.121:0/71540438 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:00.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.599+0000 7f12d8b46640 1 -- 172.21.15.121:0/71540438 >> v1:172.21.15.121:6791/0 conn(0x7f12d413bfc0 legacy=0x7f12d413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d9b48640 1 -- 172.21.15.121:0/71540438 >> v1:172.21.15.153:6792/0 conn(0x7f12d413fad0 legacy=0x7f12d4141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764128810 0 0) 0x7f12d4139a60 con 0x7f12d413bfc0 2024-09-13T13:53:00.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): my global_id is 4576 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12bc003520 con 0x7f12d413bfc0 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d9347640 1 -- 172.21.15.121:0/71540438 >> v1:172.21.15.153:6789/0 conn(0x7f12d4138c50 legacy=0x7f12d40a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4269394450 0 0) 0x7f12d409df00 con 0x7f12d413fad0 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): my global_id is 104182 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12d4139a60 con 0x7f12d413fad0 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 735945010 0 0) 0x7f12cc0014a0 con 0x7f12d4138c50 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient(hunting): my global_id is 104185 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12d409df00 con 0x7f12d4138c50 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2375122930 0 0) 0x7f12bc003520 con 0x7f12d413bfc0 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 >> v1:172.21.15.153:6792/0 conn(0x7f12d413fad0 legacy=0x7f12d4141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 >> v1:172.21.15.153:6789/0 conn(0x7f12d4138c50 legacy=0x7f12d40a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 monclient: found mon.noname-c 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d409d940 con 0x7f12d413bfc0 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: _finish_auth 0 2024-09-13T13:53:00.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c8001ae0 con 0x7f12d413bfc0 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12db5ab640 5 monclient: authenticate success, global_id 4576 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12db5ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: dump: 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.608 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12db5ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12c80048a0 con 0x7f12d413bfc0 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:00.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12db5ab640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 1 -- 172.21.15.121:0/71540438 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c8004e10 con 0x7f12d413bfc0 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12db5ab640 4 set_mon_vals no callback set 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.600+0000 7f12d3fff640 10 monclient: dump: 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:00.610 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 10 monclient: shutdown 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 1 -- 172.21.15.121:0/71540438 >> v1:172.21.15.121:6791/0 conn(0x7f12d413bfc0 legacy=0x7f12d413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 1 -- 172.21.15.121:0/71540438 shutdown_connections 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 1 -- 172.21.15.121:0/71540438 >> 172.21.15.121:0/71540438 conn(0x7f12d4069360 msgr2=0x7f12d40ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 1 -- 172.21.15.121:0/71540438 shutdown_connections 2024-09-13T13:53:00.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 1 -- 172.21.15.121:0/71540438 wait complete. 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 10 monclient: build_initial_monmap 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.601+0000 7f12db5ab640 10 monclient: monmap: 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:00.612 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 1 Processor -- start 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 1 -- start start 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient: init 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient: _add_conns ranks=[3,0,4,6,1,2,8,5,7] 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient(hunting): picked mon.noname-g con 0x7f12d4138c50 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient(hunting): picked mon.noname-a con 0x7f12d413bfc0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient(hunting): picked mon.noname-c con 0x7f12d413fad0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:00.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d40a1010 con 0x7f12d413bfc0 2024-09-13T13:53:00.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d40a12f0 con 0x7f12d4138c50 2024-09-13T13:53:00.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d413f680 con 0x7f12d413fad0 2024-09-13T13:53:00.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:00.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12db5ab640 10 monclient(hunting): authenticate will time out at 860.232740s 2024-09-13T13:53:00.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12d8b46640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f12d413bfc0 0x7f12d41d3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59120/0 (socket says 172.21.15.121:59120) 2024-09-13T13:53:00.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.602+0000 7f12d8b46640 1 -- 172.21.15.121:0/1942433112 learned_addr learned my addr 172.21.15.121:0/1942433112 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:00.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.604+0000 7f12d8b46640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.121:6789/0 conn(0x7f12d413bfc0 legacy=0x7f12d41d3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.604+0000 7f12d9b48640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.121:6791/0 conn(0x7f12d413fad0 legacy=0x7f12d41d67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d9347640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.153:6790/0 conn(0x7f12d4138c50 legacy=0x7f12d41322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3221974558 0 0) 0x7f12d40a1010 con 0x7f12d413bfc0 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): my global_id is 4680 2024-09-13T13:53:00.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12c0003550 con 0x7f12d413bfc0 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392289605 0 0) 0x7f12d413f680 con 0x7f12d413fad0 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): my global_id is 4585 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12d40a1010 con 0x7f12d413fad0 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774197557 0 0) 0x7f12d40a12f0 con 0x7f12d4138c50 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): my global_id is 4548 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12d413f680 con 0x7f12d4138c50 2024-09-13T13:53:00.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 257786063 0 0) 0x7f12d40a1010 con 0x7f12d413fad0 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12d40a12f0 con 0x7f12d413fad0 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 856601675 0 0) 0x7f12c0003550 con 0x7f12d413bfc0 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12d40a1010 con 0x7f12d413bfc0 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c4003990 con 0x7f12d413fad0 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.622 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:00.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c8004d90 con 0x7f12d413bfc0 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:00.624 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2219714747 0 0) 0x7f12d413f680 con 0x7f12d4138c50 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12c0003550 con 0x7f12d4138c50 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12cc001440 con 0x7f12d4138c50 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:00.625 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:00.626 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1509875120 0 0) 0x7f12d40a12f0 con 0x7f12d413fad0 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.153:6790/0 conn(0x7f12d4138c50 legacy=0x7f12d41322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.121:6789/0 conn(0x7f12d413bfc0 legacy=0x7f12d41d3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 monclient: found mon.c 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d40a8fa0 con 0x7f12d413fad0 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:00.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 5 monclient: authenticate success, global_id 4585 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 10 monclient: _renew_subs 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12d40a15d0 con 0x7f12d413fad0 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12c4003ea0 con 0x7f12d413fad0 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c4004410 con 0x7f12d413fad0 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d1ffb640 10 monclient: dump: 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.628 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:00.629 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 10 monclient: _renew_subs 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12d37fe640 4 set_mon_vals no callback set 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.605+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f12d409fdd0 con 0x7f12d413fad0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.606+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f12c40046b0 con 0x7f12d413fad0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.607+0000 7f12db5ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.607+0000 7f12db5ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.607+0000 7f12db5ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.607+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d4137910 con 0x7f12d413fad0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.607+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f12c404e440 con 0x7f12d413fad0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.607+0000 7f12d9347640 10 monclient: get_auth_request con 0x7f12c00404f0 auth_method 0 2024-09-13T13:53:00.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.611+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f12d4137910 con 0x7f12d413fad0 2024-09-13T13:53:00.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.611+0000 7f12d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:00.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.611+0000 7f12d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:00.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.856+0000 7f12db5ab640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.856+0000 7f12db5ab640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.856+0000 7f12db5ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.856+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f12d413dab0 con 0x7f12d413fad0 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.857+0000 7f12d1ffb640 1 -- 172.21.15.121:0/1942433112 <== mon.4 v1:172.21.15.121:6791/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+2744 (unknown 254746657 0 199096064) 0x7f12c4017a50 con 0x7f12d413fad0 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.857+0000 7f12d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.857+0000 7f12d1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:53:00.859 INFO:teuthology.orchestra.run.smithi121.stdout:{"election_epoch":28,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"d","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","modified":"2024-09-13T13:50:47.391384Z","created":"2024-09-13T13:50:47.391384Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6791","nonce":0}]},"addr":"172.21.15.121:6791/0","public_addr":"172.21.15.121:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6791","nonce":0}]},"addr":"172.21.15.153:6791/0","public_addr":"172.21.15.153:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6792","nonce":0}]},"addr":"172.21.15.121:6792/0","public_addr":"172.21.15.121:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6792","nonce":0}]},"addr":"172.21.15.153:6792/0","public_addr":"172.21.15.153:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6793","nonce":0}]},"addr":"172.21.15.121:6793/0","public_addr":"172.21.15.121:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.153:6812/1206688214 conn(0x7f12c00404f0 legacy=0x7f12c00429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 10 monclient: shutdown 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 >> v1:172.21.15.121:6791/0 conn(0x7f12d413fad0 legacy=0x7f12d41d67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 shutdown_connections 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 >> 172.21.15.121:0/1942433112 conn(0x7f12d4069360 msgr2=0x7f12d41355e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:00.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 shutdown_connections 2024-09-13T13:53:00.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.859+0000 7f12db5ab640 1 -- 172.21.15.121:0/1942433112 wait complete. 2024-09-13T13:53:00.871 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-13T13:53:00.872 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T13:53:00.882 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T13:53:00.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:53:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.978+0000 7faa58a7f640 10 monclient: get_monmap_and_config 2024-09-13T13:53:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.978+0000 7faa58a7f640 10 monclient: build_initial_monmap 2024-09-13T13:53:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.978+0000 7faa58a7f640 10 monclient: monmap: 2024-09-13T13:53:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:00.982 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:00.982 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:00.979901+0000 2024-09-13T13:53:00.982 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:00.979901+0000 2024-09-13T13:53:00.982 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:00.982 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:00.982 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:00.983 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:00.983 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:00.983 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:00.983 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:00.983 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:00.983 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:00.984 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:00.984 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:00.984 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:00.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.978+0000 7faa58a7f640 1 Processor -- start 2024-09-13T13:53:00.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.978+0000 7faa58a7f640 1 -- start start 2024-09-13T13:53:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.978+0000 7faa58a7f640 10 monclient: init 2024-09-13T13:53:00.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:00.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient: _add_conns ranks=[1,3,6,0,4,8,5,7,2] 2024-09-13T13:53:00.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient(hunting): picked mon.noname-f con 0x7faa54138c50 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:00.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient(hunting): picked mon.noname-g con 0x7faa5413bfc0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient(hunting): picked mon.noname-d con 0x7faa5413fad0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa540a5e80 con 0x7faa54138c50 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa54139a60 con 0x7faa5413bfc0 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa5409df00 con 0x7faa5413fad0 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa58a7f640 10 monclient(hunting): authenticate will time out at 860.609454s 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa52d9d640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7faa5413fad0 0x7faa54141ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51478/0 (socket says 172.21.15.121:51478) 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.979+0000 7faa52d9d640 1 -- 172.21.15.121:0/2566506801 learned_addr learned my addr 172.21.15.121:0/2566506801 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa52d9d640 1 -- 172.21.15.121:0/2566506801 >> v1:172.21.15.121:6792/0 conn(0x7faa5413fad0 legacy=0x7faa54141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492803941 0 0) 0x7faa5409df00 con 0x7faa5413fad0 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa5159a640 10 monclient(hunting): _negotiate 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa5159a640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:00.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa5159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa5159a640 10 monclient(hunting): my global_id is 4668 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.981+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa40003520 con 0x7faa5413fad0 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa51d9b640 1 -- 172.21.15.121:0/2566506801 >> v1:172.21.15.153:6790/0 conn(0x7faa5413bfc0 legacy=0x7faa5413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5259c640 1 -- 172.21.15.121:0/2566506801 >> v1:172.21.15.153:6789/0 conn(0x7faa54138c50 legacy=0x7faa540a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1046857692 0 0) 0x7faa40003520 con 0x7faa5413fad0 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 >> v1:172.21.15.153:6790/0 conn(0x7faa5413bfc0 legacy=0x7faa5413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 >> v1:172.21.15.153:6789/0 conn(0x7faa54138c50 legacy=0x7faa540a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: _finish_hunting 0 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 monclient: found mon.noname-d 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5409d940 con 0x7faa5413fad0 2024-09-13T13:53:00.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 5 monclient: authenticate success, global_id 4668 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa44001ae0 con 0x7faa5413fad0 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: dump: 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.000 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa440048a0 con 0x7faa5413fad0 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 1 -- 172.21.15.121:0/2566506801 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa44004dd0 con 0x7faa5413fad0 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 4 set_mon_vals no callback set 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa5159a640 10 monclient: dump: 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.002 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 10 monclient: shutdown 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 1 -- 172.21.15.121:0/2566506801 >> v1:172.21.15.121:6792/0 conn(0x7faa5413fad0 legacy=0x7faa54141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 1 -- 172.21.15.121:0/2566506801 shutdown_connections 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 1 -- 172.21.15.121:0/2566506801 >> 172.21.15.121:0/2566506801 conn(0x7faa54069360 msgr2=0x7faa540ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 1 -- 172.21.15.121:0/2566506801 shutdown_connections 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.982+0000 7faa58a7f640 1 -- 172.21.15.121:0/2566506801 wait complete. 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.983+0000 7faa58a7f640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.983+0000 7faa58a7f640 10 monclient: monmap: 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:01.004 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.983+0000 7faa58a7f640 1 Processor -- start 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 1 -- start start 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient: init 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient: _add_conns ranks=[6,4,3,7,8,5,0,2,1] 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient(hunting): picked mon.noname-d con 0x7faa54138c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient(hunting): picked mon.noname-c con 0x7faa5413bfc0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient(hunting): picked mon.noname-g con 0x7faa5413fad0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa540a58e0 con 0x7faa5413fad0 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa540a0fc0 con 0x7faa5413bfc0 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa540a12a0 con 0x7faa54138c50 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa58a7f640 10 monclient(hunting): authenticate will time out at 860.614487s 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa5259c640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7faa54138c50 0x7faa5409e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51488/0 (socket says 172.21.15.121:51488) 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.984+0000 7faa5259c640 1 -- 172.21.15.121:0/1518001476 learned_addr learned my addr 172.21.15.121:0/1518001476 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa5259c640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.121:6792/0 conn(0x7faa54138c50 legacy=0x7faa5409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa51d9b640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.121:6791/0 conn(0x7faa5413bfc0 legacy=0x7faa5409edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374616589 0 0) 0x7faa540a12a0 con 0x7faa54138c50 2024-09-13T13:53:01.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): my global_id is 4677 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa34003520 con 0x7faa54138c50 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256800107 0 0) 0x7faa540a0fc0 con 0x7faa5413bfc0 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 10 monclient(hunting): my global_id is 4594 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa540a12a0 con 0x7faa5413bfc0 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.986+0000 7faa52d9d640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6790/0 conn(0x7faa5413fad0 legacy=0x7faa541d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa52d9d640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6790/0 conn(0x7faa5413fad0 legacy=0x7faa541d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa52d9d640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6790/0 conn(0x7faa5413fad0 legacy=0x7faa541d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T13:53:01.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa52d9d640 1 --1- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6790/0 conn(0x7faa5413fad0 0x7faa541d60c0 :-1 s=OPENED pgs=121 cs=1 l=1).handle_message read tag failed 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa52d9d640 1 --1- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6790/0 conn(0x7faa5413fad0 0x7faa541d60c0 :-1 s=OPENED pgs=121 cs=1 l=1).fault on lossy channel, failing 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2960508569 0 0) 0x7faa34003520 con 0x7faa54138c50 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa540a0fc0 con 0x7faa54138c50 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.153:6790/0 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa4c003990 con 0x7faa54138c50 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): dump: 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.008 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:01.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3128162814 0 0) 0x7faa540a12a0 con 0x7faa5413bfc0 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa34003520 con 0x7faa5413bfc0 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa3c003610 con 0x7faa5413bfc0 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient(hunting): dump: 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.010 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1779315707 0 0) 0x7faa540a0fc0 con 0x7faa54138c50 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.121:6791/0 conn(0x7faa5413bfc0 legacy=0x7faa5409edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6790/0 conn(0x7faa5413fad0 legacy=0x7faa541d60c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-13T13:53:01.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 monclient: found mon.d 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5409f4e0 con 0x7faa54138c50 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 5 monclient: authenticate success, global_id 4677 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 10 monclient: _renew_subs 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa54139f10 con 0x7faa54138c50 2024-09-13T13:53:01.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 10 monclient: _renew_subs 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7faa5409faa0 con 0x7faa54138c50 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa4c003f20 con 0x7faa54138c50 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa4c004510 con 0x7faa54138c50 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa50d99640 4 set_mon_vals no callback set 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.987+0000 7faa4affd640 10 monclient: dump: 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.013 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7faa4c012020 con 0x7faa54138c50 2024-09-13T13:53:01.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa58a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa58a7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa58a7f640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa38005080 con 0x7faa54138c50 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7faa4c04e620 con 0x7faa54138c50 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.988+0000 7faa51d9b640 10 monclient: get_auth_request con 0x7faa34040070 auth_method 0 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.993+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7faa540a5e80 con 0x7faa54138c50 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.993+0000 7faa4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:00.993+0000 7faa4affd640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:01.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.163+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa38000b90 con 0x7faa34040070 2024-09-13T13:53:01.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.165+0000 7faa4affd640 1 -- 172.21.15.121:0/1518001476 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33967 (unknown 2965378022 0 1333185239) 0x7faa38000b90 con 0x7faa34040070 2024-09-13T13:53:01.167 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:53:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.153:6812/1206688214 conn(0x7faa34040070 legacy=0x7faa34042530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 10 monclient: shutdown 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 >> v1:172.21.15.121:6792/0 conn(0x7faa54138c50 legacy=0x7faa5409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 shutdown_connections 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 >> 172.21.15.121:0/1518001476 conn(0x7faa54069360 msgr2=0x7faa54136800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 shutdown_connections 2024-09-13T13:53:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.167+0000 7faa58a7f640 1 -- 172.21.15.121:0/1518001476 wait complete. 2024-09-13T13:53:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-13T13:52:59.234851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163668,"kb_used_data":1504,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067372,"statfs":{"total":579820584960,"available":579652988928,"internally_reserved":0,"allocated":1540096,"data_stored":1560928,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002745"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.401992+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:52:08.401992+0000","last_peered":"2024-09-13T13:52:08.401992+0000","last_clean":"2024-09-13T13:52:08.401992+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:52:08.401992+0000","last_undegraded":"2024-09-13T13:52:08.401992+0000","last_fullsized":"2024-09-13T13:52:08.401992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.169843+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:52:09.169843+0000","last_peered":"2024-09-13T13:52:09.169843+0000","last_clean":"2024-09-13T13:52:09.169843+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:52:09.169843+0000","last_undegraded":"2024-09-13T13:52:09.169843+0000","last_fullsized":"2024-09-13T13:52:09.169843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402216+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:52:08.402216+0000","last_peered":"2024-09-13T13:52:08.402216+0000","last_clean":"2024-09-13T13:52:08.402216+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:52:08.402216+0000","last_undegraded":"2024-09-13T13:52:08.402216+0000","last_fullsized":"2024-09-13T13:52:08.402216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.171472+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:52:09.171472+0000","last_peered":"2024-09-13T13:52:09.171472+0000","last_clean":"2024-09-13T13:52:09.171472+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:52:09.171472+0000","last_undegraded":"2024-09-13T13:52:09.171472+0000","last_fullsized":"2024-09-13T13:52:09.171472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":36,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403901+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:52:08.403901+0000","last_peered":"2024-09-13T13:52:08.403901+0000","last_clean":"2024-09-13T13:52:08.403901+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:52:08.403901+0000","last_undegraded":"2024-09-13T13:52:08.403901+0000","last_fullsized":"2024-09-13T13:52:08.403901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.404547+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:52:08.404547+0000","last_peered":"2024-09-13T13:52:08.404547+0000","last_clean":"2024-09-13T13:52:08.404547+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:52:08.404547+0000","last_undegraded":"2024-09-13T13:52:08.404547+0000","last_fullsized":"2024-09-13T13:52:08.404547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402141+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:52:08.402141+0000","last_peered":"2024-09-13T13:52:08.402141+0000","last_clean":"2024-09-13T13:52:08.402141+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:52:08.402141+0000","last_undegraded":"2024-09-13T13:52:08.402141+0000","last_fullsized":"2024-09-13T13:52:08.402141+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":35,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403421+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:52:08.403421+0000","last_peered":"2024-09-13T13:52:08.403421+0000","last_clean":"2024-09-13T13:52:08.403421+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:52:08.403421+0000","last_undegraded":"2024-09-13T13:52:08.403421+0000","last_fullsized":"2024-09-13T13:52:08.403421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":125,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402247+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:52:08.402247+0000","last_peered":"2024-09-13T13:52:08.402247+0000","last_clean":"2024-09-13T13:52:08.402247+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:52:08.402247+0000","last_undegraded":"2024-09-13T13:52:08.402247+0000","last_fullsized":"2024-09-13T13:52:08.402247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607571,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607572,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:53:01.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:53:01.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.277+0000 7fb0ab4db640 10 monclient: get_monmap_and_config 2024-09-13T13:53:01.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.277+0000 7fb0ab4db640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.277+0000 7fb0ab4db640 10 monclient: monmap: 2024-09-13T13:53:01.280 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.280 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.280 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:01.278905+0000 2024-09-13T13:53:01.280 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:01.278905+0000 2024-09-13T13:53:01.281 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.281 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.281 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.281 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.281 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.281 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.282 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.282 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.282 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.282 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.282 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.283 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.277+0000 7fb0ab4db640 1 Processor -- start 2024-09-13T13:53:01.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.277+0000 7fb0ab4db640 1 -- start start 2024-09-13T13:53:01.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.277+0000 7fb0ab4db640 10 monclient: init 2024-09-13T13:53:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient: _add_conns ranks=[4,8,6,7,2,3,1,0,5] 2024-09-13T13:53:01.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient(hunting): picked mon.noname-c con 0x7fb0a4138c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:01.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient(hunting): picked mon.noname-e con 0x7fb0a413bfc0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:01.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient(hunting): picked mon.noname-d con 0x7fb0a413fad0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:01.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a40a5e80 con 0x7fb0a4138c50 2024-09-13T13:53:01.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a4139a60 con 0x7fb0a413fad0 2024-09-13T13:53:01.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a409df00 con 0x7fb0a413bfc0 2024-09-13T13:53:01.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0ab4db640 10 monclient(hunting): authenticate will time out at 860.908586s 2024-09-13T13:53:01.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0a9277640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fb0a4138c50 0x7fb0a40a5770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33652/0 (socket says 172.21.15.121:33652) 2024-09-13T13:53:01.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.278+0000 7fb0a9277640 1 -- 172.21.15.121:0/1198142663 learned_addr learned my addr 172.21.15.121:0/1198142663 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb0a8a76640 1 -- 172.21.15.121:0/1198142663 >> v1:172.21.15.121:6793/0 conn(0x7fb0a413bfc0 legacy=0x7fb0a413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb0a9a78640 1 -- 172.21.15.121:0/1198142663 >> v1:172.21.15.121:6792/0 conn(0x7fb0a413fad0 legacy=0x7fb0a4141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb0a9277640 1 -- 172.21.15.121:0/1198142663 >> v1:172.21.15.121:6791/0 conn(0x7fb0a4138c50 legacy=0x7fb0a40a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036653149 0 0) 0x7fb0a409df00 con 0x7fb0a413bfc0 2024-09-13T13:53:01.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb097fff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb097fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb097fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.280+0000 7fb097fff640 10 monclient(hunting): my global_id is 4634 2024-09-13T13:53:01.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb090003520 con 0x7fb0a413bfc0 2024-09-13T13:53:01.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872366371 0 0) 0x7fb0a4139a60 con 0x7fb0a413fad0 2024-09-13T13:53:01.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): my global_id is 4686 2024-09-13T13:53:01.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a409df00 con 0x7fb0a413fad0 2024-09-13T13:53:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468633811 0 0) 0x7fb09c001450 con 0x7fb0a4138c50 2024-09-13T13:53:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient(hunting): my global_id is 4603 2024-09-13T13:53:01.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a4139a60 con 0x7fb0a4138c50 2024-09-13T13:53:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2024125536 0 0) 0x7fb090003520 con 0x7fb0a413bfc0 2024-09-13T13:53:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 >> v1:172.21.15.121:6792/0 conn(0x7fb0a413fad0 legacy=0x7fb0a4141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 >> v1:172.21.15.121:6791/0 conn(0x7fb0a4138c50 legacy=0x7fb0a40a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 monclient: found mon.noname-e 2024-09-13T13:53:01.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:53:01.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a409d940 con 0x7fb0a413bfc0 2024-09-13T13:53:01.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb098001eb0 con 0x7fb0a413bfc0 2024-09-13T13:53:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: dump: 2024-09-13T13:53:01.306 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.307 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.308 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.309 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.309 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.309 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 5 monclient: authenticate success, global_id 4634 2024-09-13T13:53:01.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:01.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb098003990 con 0x7fb0a413bfc0 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 1 -- 172.21.15.121:0/1198142663 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb098003f00 con 0x7fb0a413bfc0 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 4 set_mon_vals no callback set 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:01.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb097fff640 10 monclient: dump: 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.311 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 10 monclient: shutdown 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 1 -- 172.21.15.121:0/1198142663 >> v1:172.21.15.121:6793/0 conn(0x7fb0a413bfc0 legacy=0x7fb0a413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 1 -- 172.21.15.121:0/1198142663 shutdown_connections 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 1 -- 172.21.15.121:0/1198142663 >> 172.21.15.121:0/1198142663 conn(0x7fb0a4069360 msgr2=0x7fb0a40ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.281+0000 7fb0ab4db640 1 -- 172.21.15.121:0/1198142663 shutdown_connections 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.282+0000 7fb0ab4db640 1 -- 172.21.15.121:0/1198142663 wait complete. 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.282+0000 7fb0ab4db640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.282+0000 7fb0ab4db640 10 monclient: monmap: 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.313 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.282+0000 7fb0ab4db640 1 Processor -- start 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.282+0000 7fb0ab4db640 1 -- start start 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.282+0000 7fb0ab4db640 10 monclient: init 2024-09-13T13:53:01.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient: _add_conns ranks=[2,5,6,3,1,0,7,8,4] 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient(hunting): picked mon.noname-b con 0x7fb0a4138c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient(hunting): picked mon.noname-h con 0x7fb0a413bfc0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient(hunting): picked mon.noname-d con 0x7fb0a413fad0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a40a0d80 con 0x7fb0a4138c50 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a41cea70 con 0x7fb0a413bfc0 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a40a1060 con 0x7fb0a413fad0 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0ab4db640 10 monclient(hunting): authenticate will time out at 860.913480s 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0a9277640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fb0a4138c50 0x7fb0a41cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55074/0 (socket says 172.21.15.121:55074) 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.283+0000 7fb0a9277640 1 -- 172.21.15.121:0/860109021 learned_addr learned my addr 172.21.15.121:0/860109021 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.285+0000 7fb0a9277640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.121:6790/0 conn(0x7fb0a4138c50 legacy=0x7fb0a41cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.285+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 DispatchQueue::entry inject delay of 0.002000 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.285+0000 7fb0a9a78640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.121:6792/0 conn(0x7fb0a413fad0 legacy=0x7fb0a41d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.285+0000 7fb0a8a76640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.153:6791/0 conn(0x7fb0a413bfc0 legacy=0x7fb0a41ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682607762 0 0) 0x7fb0a40a0d80 con 0x7fb0a4138c50 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): my global_id is 4619 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb088003520 con 0x7fb0a4138c50 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 471374825 0 0) 0x7fb0a41cea70 con 0x7fb0a413bfc0 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): my global_id is 4613 2024-09-13T13:53:01.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a40a0d80 con 0x7fb0a413bfc0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802917273 0 0) 0x7fb0a40a1060 con 0x7fb0a413fad0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 10 monclient(hunting): my global_id is 4695 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0a41cea70 con 0x7fb0a413fad0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.287+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 79574220 0 0) 0x7fb0a41cea70 con 0x7fb0a413fad0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0a40a1060 con 0x7fb0a413fad0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0a0003610 con 0x7fb0a413fad0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:01.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.318 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1848439811 0 0) 0x7fb088003520 con 0x7fb0a4138c50 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0a41cea70 con 0x7fb0a4138c50 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb09c0047e0 con 0x7fb0a4138c50 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.319 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3129535270 0 0) 0x7fb0a40a1060 con 0x7fb0a413fad0 2024-09-13T13:53:01.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.153:6791/0 conn(0x7fb0a413bfc0 legacy=0x7fb0a41ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.121:6790/0 conn(0x7fb0a4138c50 legacy=0x7fb0a41cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 monclient: found mon.d 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a4138360 con 0x7fb0a413fad0 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 5 monclient: authenticate success, global_id 4695 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 10 monclient: _renew_subs 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a40a0aa0 con 0x7fb0a413fad0 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0a0003a60 con 0x7fb0a413fad0 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0a0003f90 con 0x7fb0a413fad0 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb095ffb640 10 monclient: dump: 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.322 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0977fe640 4 set_mon_vals no callback set 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 10 monclient: _renew_subs 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.288+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0a413e390 con 0x7fb0a413fad0 2024-09-13T13:53:01.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.289+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fb0a0004130 con 0x7fb0a413fad0 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.289+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fb0a004e040 con 0x7fb0a413fad0 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.289+0000 7fb0ab4db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.289+0000 7fb0ab4db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.289+0000 7fb0ab4db640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.289+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a4134ce0 con 0x7fb0a413fad0 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.290+0000 7fb0a9277640 10 monclient: get_auth_request con 0x7fb088040100 auth_method 0 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.293+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fb0a0017650 con 0x7fb0a413fad0 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.293+0000 7fb095ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.293+0000 7fb095ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:01.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.473+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb0a4138790 con 0x7fb088040100 2024-09-13T13:53:01.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.474+0000 7fb095ffb640 1 -- 172.21.15.121:0/860109021 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33967 (unknown 2965378022 0 3197077879) 0x7fb0a4138790 con 0x7fb088040100 2024-09-13T13:53:01.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:53:01.477 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:53:01.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.153:6812/1206688214 conn(0x7fb088040100 legacy=0x7fb0880425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 10 monclient: shutdown 2024-09-13T13:53:01.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 >> v1:172.21.15.121:6792/0 conn(0x7fb0a413fad0 legacy=0x7fb0a41d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 shutdown_connections 2024-09-13T13:53:01.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 >> 172.21.15.121:0/860109021 conn(0x7fb0a4069360 msgr2=0x7fb0a4139d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 shutdown_connections 2024-09-13T13:53:01.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.476+0000 7fb0ab4db640 1 -- 172.21.15.121:0/860109021 wait complete. 2024-09-13T13:53:01.489 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T13:53:01.235320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163668,"kb_used_data":1504,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067372,"statfs":{"total":579820584960,"available":579652988928,"internally_reserved":0,"allocated":1540096,"data_stored":1560928,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.401992+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:52:08.401992+0000","last_peered":"2024-09-13T13:52:08.401992+0000","last_clean":"2024-09-13T13:52:08.401992+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:52:08.401992+0000","last_undegraded":"2024-09-13T13:52:08.401992+0000","last_fullsized":"2024-09-13T13:52:08.401992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.169843+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:52:09.169843+0000","last_peered":"2024-09-13T13:52:09.169843+0000","last_clean":"2024-09-13T13:52:09.169843+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:52:09.169843+0000","last_undegraded":"2024-09-13T13:52:09.169843+0000","last_fullsized":"2024-09-13T13:52:09.169843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402216+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:52:08.402216+0000","last_peered":"2024-09-13T13:52:08.402216+0000","last_clean":"2024-09-13T13:52:08.402216+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:52:08.402216+0000","last_undegraded":"2024-09-13T13:52:08.402216+0000","last_fullsized":"2024-09-13T13:52:08.402216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.171472+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:52:09.171472+0000","last_peered":"2024-09-13T13:52:09.171472+0000","last_clean":"2024-09-13T13:52:09.171472+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:52:09.171472+0000","last_undegraded":"2024-09-13T13:52:09.171472+0000","last_fullsized":"2024-09-13T13:52:09.171472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":36,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403901+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:52:08.403901+0000","last_peered":"2024-09-13T13:52:08.403901+0000","last_clean":"2024-09-13T13:52:08.403901+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:52:08.403901+0000","last_undegraded":"2024-09-13T13:52:08.403901+0000","last_fullsized":"2024-09-13T13:52:08.403901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.404547+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:52:08.404547+0000","last_peered":"2024-09-13T13:52:08.404547+0000","last_clean":"2024-09-13T13:52:08.404547+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:52:08.404547+0000","last_undegraded":"2024-09-13T13:52:08.404547+0000","last_fullsized":"2024-09-13T13:52:08.404547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402141+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:52:08.402141+0000","last_peered":"2024-09-13T13:52:08.402141+0000","last_clean":"2024-09-13T13:52:08.402141+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:52:08.402141+0000","last_undegraded":"2024-09-13T13:52:08.402141+0000","last_fullsized":"2024-09-13T13:52:08.402141+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":35,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403421+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:52:08.403421+0000","last_peered":"2024-09-13T13:52:08.403421+0000","last_clean":"2024-09-13T13:52:08.403421+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:52:08.403421+0000","last_undegraded":"2024-09-13T13:52:08.403421+0000","last_fullsized":"2024-09-13T13:52:08.403421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":125,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402247+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:52:08.402247+0000","last_peered":"2024-09-13T13:52:08.402247+0000","last_clean":"2024-09-13T13:52:08.402247+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:52:08.402247+0000","last_undegraded":"2024-09-13T13:52:08.402247+0000","last_fullsized":"2024-09-13T13:52:08.402247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607571,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607572,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:53:01.490 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T13:53:01.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.586+0000 7f1db3e73640 10 monclient: get_monmap_and_config 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.586+0000 7f1db3e73640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.586+0000 7f1db3e73640 10 monclient: monmap: 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:01.587854+0000 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:01.587854+0000 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.588 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.589 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.586+0000 7f1db3e73640 1 Processor -- start 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.586+0000 7f1db3e73640 1 -- start start 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.586+0000 7f1db3e73640 10 monclient: init 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient: _add_conns ranks=[2,0,7,5,6,1,3,8,4] 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient(hunting): picked mon.noname-b con 0x7f1dac138c50 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:01.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient(hunting): picked mon.noname-a con 0x7f1dac13bfc0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient(hunting): picked mon.noname-i con 0x7f1dac13fad0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac0a5e80 con 0x7f1dac13bfc0 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac139a60 con 0x7f1dac138c50 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac09df00 con 0x7f1dac13fad0 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db3e73640 10 monclient(hunting): authenticate will time out at 861.217485s 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db140e640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1dac13bfc0 0x7f1dac13e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59130/0 (socket says 172.21.15.121:59130) 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db140e640 1 -- 172.21.15.121:0/1207957201 learned_addr learned my addr 172.21.15.121:0/1207957201 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.587+0000 7f1db1c0f640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f1dac138c50 0x7f1dac0a5770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55082/0 (socket says 172.21.15.121:55082) 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db140e640 1 -- 172.21.15.121:0/1207957201 >> v1:172.21.15.121:6789/0 conn(0x7f1dac13bfc0 legacy=0x7f1dac13e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db1c0f640 1 -- 172.21.15.121:0/1207957201 >> v1:172.21.15.121:6790/0 conn(0x7f1dac138c50 legacy=0x7f1dac0a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730300261 0 0) 0x7f1dac139a60 con 0x7f1dac138c50 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): my global_id is 4628 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d98003520 con 0x7f1dac138c50 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778116266 0 0) 0x7f1d9c001450 con 0x7f1dac13bfc0 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 10 monclient(hunting): my global_id is 4689 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dac139a60 con 0x7f1dac13bfc0 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.589+0000 7f1db2410640 1 -- 172.21.15.121:0/1207957201 >> v1:172.21.15.153:6792/0 conn(0x7f1dac13fad0 legacy=0x7f1dac141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3548854161 0 0) 0x7f1d98003520 con 0x7f1dac138c50 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 >> v1:172.21.15.153:6792/0 conn(0x7f1dac13fad0 legacy=0x7f1dac141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 >> v1:172.21.15.121:6789/0 conn(0x7f1dac13bfc0 legacy=0x7f1dac13e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 monclient: found mon.noname-b 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac09d940 con 0x7f1dac138c50 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1da8001a10 con 0x7f1dac138c50 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 5 monclient: authenticate success, global_id 4628 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: dump: 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.597 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079987537 0 0) 0x7f1dac09df00 con 0x7f1dac13fad0 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1da8001ce0 con 0x7f1dac138c50 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 1 -- 172.21.15.121:0/1207957201 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1da8004b20 con 0x7f1dac138c50 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 4 set_mon_vals no callback set 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db0c0d640 10 monclient: dump: 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.599 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 10 monclient: shutdown 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 1 -- 172.21.15.121:0/1207957201 >> v1:172.21.15.121:6790/0 conn(0x7f1dac138c50 legacy=0x7f1dac0a5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 1 -- 172.21.15.121:0/1207957201 shutdown_connections 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 1 -- 172.21.15.121:0/1207957201 >> 172.21.15.121:0/1207957201 conn(0x7f1dac069360 msgr2=0x7f1dac0ab800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 1 -- 172.21.15.121:0/1207957201 shutdown_connections 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.590+0000 7f1db3e73640 1 -- 172.21.15.121:0/1207957201 wait complete. 2024-09-13T13:53:01.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.591+0000 7f1db3e73640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.591+0000 7f1db3e73640 10 monclient: monmap: 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.601 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.602 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.591+0000 7f1db3e73640 1 Processor -- start 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.591+0000 7f1db3e73640 1 -- start start 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.591+0000 7f1db3e73640 10 monclient: init 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient: _add_conns ranks=[4,5,0,3,2,6,1,8,7] 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient(hunting): picked mon.noname-c con 0x7f1dac138c50 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient(hunting): picked mon.noname-h con 0x7f1dac13bfc0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient(hunting): picked mon.noname-a con 0x7f1dac13fad0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac132d70 con 0x7f1dac13fad0 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac0a5bf0 con 0x7f1dac138c50 2024-09-13T13:53:01.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dac13f4f0 con 0x7f1dac13bfc0 2024-09-13T13:53:01.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db3e73640 10 monclient(hunting): authenticate will time out at 861.222302s 2024-09-13T13:53:01.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db2410640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f1dac13fad0 0x7f1dac1d60c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:59144/0 (socket says 172.21.15.121:59144) 2024-09-13T13:53:01.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db1c0f640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f1dac138c50 0x7f1dac131f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33656/0 (socket says 172.21.15.121:33656) 2024-09-13T13:53:01.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.592+0000 7f1db2410640 1 -- 172.21.15.121:0/1653945227 learned_addr learned my addr 172.21.15.121:0/1653945227 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1db1c0f640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.121:6791/0 conn(0x7f1dac138c50 legacy=0x7f1dac131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1db2410640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.121:6789/0 conn(0x7f1dac13fad0 legacy=0x7f1dac1d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194594705 0 0) 0x7f1dac0a5bf0 con 0x7f1dac138c50 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): my global_id is 4612 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1db140e640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.153:6791/0 conn(0x7f1dac13bfc0 legacy=0x7f1dac132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d90003520 con 0x7f1dac138c50 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143417701 0 0) 0x7f1dac132d70 con 0x7f1dac13fad0 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): my global_id is 4698 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dac0a5bf0 con 0x7f1dac13fad0 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3267199903 0 0) 0x7f1dac13f4f0 con 0x7f1dac13bfc0 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): my global_id is 4622 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dac132d70 con 0x7f1dac13bfc0 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4195977770 0 0) 0x7f1dac0a5bf0 con 0x7f1dac13fad0 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dac13f4f0 con 0x7f1dac13fad0 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2437248667 0 0) 0x7f1d90003520 con 0x7f1dac138c50 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1dac0a5bf0 con 0x7f1dac138c50 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1da4003990 con 0x7f1dac13fad0 2024-09-13T13:53:01.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.594+0000 7f1da27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): dump: 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.610 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1da8001a30 con 0x7f1dac138c50 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): dump: 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.611 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3469334217 0 0) 0x7f1dac132d70 con 0x7f1dac13bfc0 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d90003520 con 0x7f1dac13bfc0 2024-09-13T13:53:01.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1d9c001b10 con 0x7f1dac13bfc0 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient(hunting): dump: 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.613 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3692891934 0 0) 0x7f1dac13f4f0 con 0x7f1dac13fad0 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.153:6791/0 conn(0x7f1dac13bfc0 legacy=0x7f1dac132660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.121:6791/0 conn(0x7f1dac138c50 legacy=0x7f1dac131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 monclient: found mon.a 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dac135ff0 con 0x7f1dac13fad0 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 5 monclient: authenticate success, global_id 4698 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 10 monclient: _renew_subs 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1dac0a1070 con 0x7f1dac13fad0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 10 monclient: _renew_subs 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1dac0a5560 con 0x7f1dac13fad0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1da4003f10 con 0x7f1dac13fad0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f1da4004400 con 0x7f1dac13fad0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da3fff640 4 set_mon_vals no callback set 2024-09-13T13:53:01.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.595+0000 7f1da27fc640 10 monclient: dump: 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.616 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f1da40045a0 con 0x7f1dac13fad0 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1db3e73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1db3e73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1db3e73640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f1da404e530 con 0x7f1dac13fad0 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d94005080 con 0x7f1dac13fad0 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.596+0000 7f1db1c0f640 10 monclient: get_auth_request con 0x7f1d90040420 auth_method 0 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.601+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f1da4017b60 con 0x7f1dac13fad0 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.601+0000 7f1da27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.601+0000 7f1da27fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:01.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.774+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1d94000b90 con 0x7f1d90040420 2024-09-13T13:53:01.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.775+0000 7f1da27fc640 1 -- 172.21.15.121:0/1653945227 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33967 (unknown 2965378022 0 3197077879) 0x7f1d94000b90 con 0x7f1d90040420 2024-09-13T13:53:01.782 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:53:01.782 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.777+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.153:6812/1206688214 conn(0x7f1d90040420 legacy=0x7f1d900428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.777+0000 7f1db3e73640 10 monclient: shutdown 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.777+0000 7f1db3e73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.777+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 >> v1:172.21.15.121:6789/0 conn(0x7f1dac13fad0 legacy=0x7f1dac1d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.778+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 shutdown_connections 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.778+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 >> 172.21.15.121:0/1653945227 conn(0x7f1dac069360 msgr2=0x7f1dac139d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.778+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 shutdown_connections 2024-09-13T13:53:01.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.778+0000 7f1db3e73640 1 -- 172.21.15.121:0/1653945227 wait complete. 2024-09-13T13:53:01.790 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T13:53:01.235320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163668,"kb_used_data":1504,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067372,"statfs":{"total":579820584960,"available":579652988928,"internally_reserved":0,"allocated":1540096,"data_stored":1560928,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002669"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.401992+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:52:08.401992+0000","last_peered":"2024-09-13T13:52:08.401992+0000","last_clean":"2024-09-13T13:52:08.401992+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:52:08.401992+0000","last_undegraded":"2024-09-13T13:52:08.401992+0000","last_fullsized":"2024-09-13T13:52:08.401992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.169843+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:52:09.169843+0000","last_peered":"2024-09-13T13:52:09.169843+0000","last_clean":"2024-09-13T13:52:09.169843+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:52:09.169843+0000","last_undegraded":"2024-09-13T13:52:09.169843+0000","last_fullsized":"2024-09-13T13:52:09.169843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402216+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:52:08.402216+0000","last_peered":"2024-09-13T13:52:08.402216+0000","last_clean":"2024-09-13T13:52:08.402216+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:52:08.402216+0000","last_undegraded":"2024-09-13T13:52:08.402216+0000","last_fullsized":"2024-09-13T13:52:08.402216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.171472+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:52:09.171472+0000","last_peered":"2024-09-13T13:52:09.171472+0000","last_clean":"2024-09-13T13:52:09.171472+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:52:09.171472+0000","last_undegraded":"2024-09-13T13:52:09.171472+0000","last_fullsized":"2024-09-13T13:52:09.171472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":36,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403901+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:52:08.403901+0000","last_peered":"2024-09-13T13:52:08.403901+0000","last_clean":"2024-09-13T13:52:08.403901+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:52:08.403901+0000","last_undegraded":"2024-09-13T13:52:08.403901+0000","last_fullsized":"2024-09-13T13:52:08.403901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.404547+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:52:08.404547+0000","last_peered":"2024-09-13T13:52:08.404547+0000","last_clean":"2024-09-13T13:52:08.404547+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:52:08.404547+0000","last_undegraded":"2024-09-13T13:52:08.404547+0000","last_fullsized":"2024-09-13T13:52:08.404547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402141+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:52:08.402141+0000","last_peered":"2024-09-13T13:52:08.402141+0000","last_clean":"2024-09-13T13:52:08.402141+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:52:08.402141+0000","last_undegraded":"2024-09-13T13:52:08.402141+0000","last_fullsized":"2024-09-13T13:52:08.402141+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":35,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403421+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:52:08.403421+0000","last_peered":"2024-09-13T13:52:08.403421+0000","last_clean":"2024-09-13T13:52:08.403421+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:52:08.403421+0000","last_undegraded":"2024-09-13T13:52:08.403421+0000","last_fullsized":"2024-09-13T13:52:08.403421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":125,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402247+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:52:08.402247+0000","last_peered":"2024-09-13T13:52:08.402247+0000","last_clean":"2024-09-13T13:52:08.402247+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:52:08.402247+0000","last_undegraded":"2024-09-13T13:52:08.402247+0000","last_fullsized":"2024-09-13T13:52:08.402247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607571,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640277,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607572,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:53:01.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T13:53:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.886+0000 7f5849b53640 10 monclient: get_monmap_and_config 2024-09-13T13:53:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.886+0000 7f5849b53640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.886+0000 7f5849b53640 10 monclient: monmap: 2024-09-13T13:53:01.889 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.889 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.889 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:01.888500+0000 2024-09-13T13:53:01.889 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:01.888500+0000 2024-09-13T13:53:01.889 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.893 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.887+0000 7f5849b53640 1 Processor -- start 2024-09-13T13:53:01.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.887+0000 7f5849b53640 1 -- start start 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.887+0000 7f5849b53640 10 monclient: init 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.887+0000 7f5849b53640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.887+0000 7f5849b53640 10 monclient: _add_conns ranks=[1,6,7,2,5,0,4,8,3] 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 10 monclient(hunting): picked mon.noname-f con 0x7f584409c950 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 10 monclient(hunting): picked mon.noname-d con 0x7f5844135b50 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 10 monclient(hunting): picked mon.noname-i con 0x7f584413f660 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5844135f50 con 0x7f584409c950 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584409d860 con 0x7f5844135b50 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584409db40 con 0x7f584413f660 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5849b53640 10 monclient(hunting): authenticate will time out at 861.518075s 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5842ffd640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f5844135b50 0x7f584413df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51506/0 (socket says 172.21.15.121:51506) 2024-09-13T13:53:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.888+0000 7f5842ffd640 1 -- 172.21.15.121:0/747286195 learned_addr learned my addr 172.21.15.121:0/747286195 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5842ffd640 1 -- 172.21.15.121:0/747286195 >> v1:172.21.15.121:6792/0 conn(0x7f5844135b50 legacy=0x7f584413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2269309403 0 0) 0x7f584409d860 con 0x7f5844135b50 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient(hunting): my global_id is 4704 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f582c003520 con 0x7f5844135b50 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5843fff640 1 -- 172.21.15.121:0/747286195 >> v1:172.21.15.153:6792/0 conn(0x7f584413f660 legacy=0x7f5844141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58437fe640 1 -- 172.21.15.121:0/747286195 >> v1:172.21.15.153:6789/0 conn(0x7f584409c950 legacy=0x7f58440a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1204966055 0 0) 0x7f582c003520 con 0x7f5844135b50 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 >> v1:172.21.15.153:6792/0 conn(0x7f584413f660 legacy=0x7f5844141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 >> v1:172.21.15.153:6789/0 conn(0x7f584409c950 legacy=0x7f58440a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 monclient: found mon.noname-d 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584409de20 con 0x7f5844135b50 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5838001ae0 con 0x7f5844135b50 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5849b53640 5 monclient: authenticate success, global_id 4704 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5849b53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: dump: 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.903 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.904 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5849b53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58380048a0 con 0x7f5844135b50 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 1 -- 172.21.15.121:0/747286195 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5838004e10 con 0x7f5844135b50 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5849b53640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f5849b53640 4 set_mon_vals no callback set 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.890+0000 7f58427fc640 10 monclient: dump: 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.905 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 10 monclient: shutdown 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 1 -- 172.21.15.121:0/747286195 >> v1:172.21.15.121:6792/0 conn(0x7f5844135b50 legacy=0x7f584413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 1 -- 172.21.15.121:0/747286195 shutdown_connections 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 1 -- 172.21.15.121:0/747286195 >> 172.21.15.121:0/747286195 conn(0x7f58440ab760 msgr2=0x7f58440abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 1 -- 172.21.15.121:0/747286195 shutdown_connections 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 1 -- 172.21.15.121:0/747286195 wait complete. 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 10 monclient: build_initial_monmap 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.891+0000 7f5849b53640 10 monclient: monmap: 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:01.907 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 1 Processor -- start 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 1 -- start start 2024-09-13T13:53:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient: init 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient: _add_conns ranks=[2,6,8,0,5,7,1,3,4] 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient(hunting): picked mon.noname-b con 0x7f584409c950 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient(hunting): picked mon.noname-d con 0x7f5844135b50 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient(hunting): picked mon.noname-e con 0x7f584413f660 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584413e440 con 0x7f584409c950 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584413e720 con 0x7f5844135b50 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584413ef40 con 0x7f584413f660 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.892+0000 7f5849b53640 10 monclient(hunting): authenticate will time out at 861.522954s 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.893+0000 7f5842ffd640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f5844135b50 0x7f584413b3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:51510/0 (socket says 172.21.15.121:51510) 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.893+0000 7f58437fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f584409c950 0x7f584413acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:55092/0 (socket says 172.21.15.121:55092) 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.893+0000 7f5842ffd640 1 -- 172.21.15.121:0/3411969720 learned_addr learned my addr 172.21.15.121:0/3411969720 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:01.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f58437fe640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.121:6790/0 conn(0x7f584409c950 legacy=0x7f584413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5842ffd640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.121:6792/0 conn(0x7f5844135b50 legacy=0x7f584413b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5843fff640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.121:6793/0 conn(0x7f584413f660 legacy=0x7f58441d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802538988 0 0) 0x7f584413e720 con 0x7f5844135b50 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): my global_id is 4713 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5830003520 con 0x7f5844135b50 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10206054 0 0) 0x7f584413e440 con 0x7f584409c950 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): my global_id is 4637 2024-09-13T13:53:01.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f584413e720 con 0x7f584409c950 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2201349111 0 0) 0x7f584413ef40 con 0x7f584413f660 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): my global_id is 4643 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f584413e440 con 0x7f584413f660 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3876282054 0 0) 0x7f5830003520 con 0x7f5844135b50 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f584413ef40 con 0x7f5844135b50 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5838001a60 con 0x7f5844135b50 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): dump: 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3380589694 0 0) 0x7f584413e440 con 0x7f584413f660 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5830003520 con 0x7f584413f660 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2685350706 0 0) 0x7f584413e720 con 0x7f584409c950 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f584413e440 con 0x7f584409c950 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f583c001ff0 con 0x7f584413f660 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): dump: 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.913 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5834003610 con 0x7f584409c950 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:01.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient(hunting): dump: 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.915 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 103050585 0 0) 0x7f584413ef40 con 0x7f5844135b50 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.121:6793/0 conn(0x7f584413f660 legacy=0x7f58441d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.121:6790/0 conn(0x7f584409c950 legacy=0x7f584413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 monclient: found mon.d 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5844142e30 con 0x7f5844135b50 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient: _finish_auth 0 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f582bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:01.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 5 monclient: authenticate success, global_id 4713 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 10 monclient: _renew_subs 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5844142a70 con 0x7f5844135b50 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 10 monclient: _renew_subs 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.895+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f58440a4e50 con 0x7f5844135b50 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5838001320 con 0x7f5844135b50 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f58380048a0 con 0x7f5844135b50 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f5841ffb640 4 set_mon_vals no callback set 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 10 monclient: dump: 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:01.917 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:01.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f5838011b20 con 0x7f5844135b50 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f5849b53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f5849b53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f5849b53640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.896+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5824005080 con 0x7f5844135b50 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.897+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f583804e2d0 con 0x7f5844135b50 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.897+0000 7f58437fe640 10 monclient: get_auth_request con 0x7f5830040770 auth_method 0 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.901+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f5838017940 con 0x7f5844135b50 2024-09-13T13:53:01.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.901+0000 7f582bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:01.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:01.901+0000 7f582bfff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.077+0000 7f5849b53640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:53:02.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.077+0000 7f5849b53640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:53:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.077+0000 7f5849b53640 10 monclient: _send_mon_message to mon.d at v1:172.21.15.121:6792/0 2024-09-13T13:53:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.077+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 --> v1:172.21.15.121:6792/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5824005370 con 0x7f5844135b50 2024-09-13T13:53:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.078+0000 7f582bfff640 1 -- 172.21.15.121:0/3411969720 <== mon.6 v1:172.21.15.121:6792/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+10893 (unknown 3600925931 0 463707345) 0x7f5838017240 con 0x7f5844135b50 2024-09-13T13:53:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.078+0000 7f582bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-13T13:53:02.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.078+0000 7f582bfff640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:53:02.081 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:53:02.081 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":27,"fsid":"21ebbdf6-8985-4682-9234-cdd2d2c1c2db","created":"2024-09-13T13:51:21.408933+0000","modified":"2024-09-13T13:52:08.391236+0000","last_up_change":"2024-09-13T13:51:36.267919+0000","last_in_change":"2024-09-13T13:51:25.134237+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T13:51:35.278046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-13T13:51:40.243263+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6c3c9e1-ad77-4e07-a73a-95b4956715e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6808","nonce":3060725185}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6809","nonce":3060725185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6811","nonce":3060725185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6810","nonce":3060725185}]},"public_addr":"172.21.15.121:6808/3060725185","cluster_addr":"172.21.15.121:6809/3060725185","heartbeat_back_addr":"172.21.15.121:6811/3060725185","heartbeat_front_addr":"172.21.15.121:6810/3060725185","state":["exists","up"]},{"osd":1,"uuid":"6d5efbfa-eca0-4806-9229-69bbc58aa605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6800","nonce":1651307734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6801","nonce":1651307734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6803","nonce":1651307734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6802","nonce":1651307734}]},"public_addr":"172.21.15.121:6800/1651307734","cluster_addr":"172.21.15.121:6801/1651307734","heartbeat_back_addr":"172.21.15.121:6803/1651307734","heartbeat_front_addr":"172.21.15.121:6802/1651307734","state":["exists","up"]},{"osd":2,"uuid":"fba721b0-d5d8-4735-b8e2-a7cf81067ae1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6804","nonce":2141781943}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6805","nonce":2141781943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6807","nonce":2141781943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.121:6806","nonce":2141781943}]},"public_addr":"172.21.15.121:6804/2141781943","cluster_addr":"172.21.15.121:6805/2141781943","heartbeat_back_addr":"172.21.15.121:6807/2141781943","heartbeat_front_addr":"172.21.15.121:6806/2141781943","state":["exists","up"]},{"osd":3,"uuid":"9bbad653-ff80-4a41-a571-666779269a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6808","nonce":934815702}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6809","nonce":934815702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6811","nonce":934815702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6810","nonce":934815702}]},"public_addr":"172.21.15.153:6808/934815702","cluster_addr":"172.21.15.153:6809/934815702","heartbeat_back_addr":"172.21.15.153:6811/934815702","heartbeat_front_addr":"172.21.15.153:6810/934815702","state":["exists","up"]},{"osd":4,"uuid":"9f1f8ec6-019a-49ae-a845-dbef0966f37d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6800","nonce":1083052486}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6801","nonce":1083052486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6803","nonce":1083052486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6802","nonce":1083052486}]},"public_addr":"172.21.15.153:6800/1083052486","cluster_addr":"172.21.15.153:6801/1083052486","heartbeat_back_addr":"172.21.15.153:6803/1083052486","heartbeat_front_addr":"172.21.15.153:6802/1083052486","state":["exists","up"]},{"osd":5,"uuid":"ba16c496-ba8f-4f30-978f-ec8c36f2656d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6804","nonce":3215471673}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6805","nonce":3215471673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6807","nonce":3215471673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.153:6806","nonce":3215471673}]},"public_addr":"172.21.15.153:6804/3215471673","cluster_addr":"172.21.15.153:6805/3215471673","heartbeat_back_addr":"172.21.15.153:6807/3215471673","heartbeat_front_addr":"172.21.15.153:6806/3215471673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.867807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.618229+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.154306+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.400091+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:30.983001+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T13:51:31.211095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.080+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.153:6812/1206688214 conn(0x7f5830040770 legacy=0x7f5830042c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.080+0000 7f5849b53640 10 monclient: shutdown 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.080+0000 7f5849b53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.081+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 >> v1:172.21.15.121:6792/0 conn(0x7f5844135b50 legacy=0x7f584413b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.081+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 shutdown_connections 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.081+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 >> 172.21.15.121:0/3411969720 conn(0x7f58440ab760 msgr2=0x7f58441338c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.081+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 shutdown_connections 2024-09-13T13:53:02.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:02.081+0000 7f5849b53640 1 -- 172.21.15.121:0/3411969720 wait complete. 2024-09-13T13:53:03.094 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T13:53:03.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:53:03.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.198+0000 7fdee35bb640 10 monclient: get_monmap_and_config 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.198+0000 7fdee35bb640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.198+0000 7fdee35bb640 10 monclient: monmap: 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:03.199860+0000 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:03.199860+0000 2024-09-13T13:53:03.200 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.201 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.202 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.202 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.202 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.202 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.202 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.198+0000 7fdee35bb640 1 Processor -- start 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.198+0000 7fdee35bb640 1 -- start start 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.198+0000 7fdee35bb640 10 monclient: init 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient: _add_conns ranks=[4,3,6,8,0,5,7,1,2] 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient(hunting): picked mon.noname-c con 0x7fdedc0a5fc0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient(hunting): picked mon.noname-g con 0x7fdedc0a6ab0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient(hunting): picked mon.noname-d con 0x7fdedc09dcf0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc09d030 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc09d310 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc09e180 con 0x7fdedc09dcf0 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee35bb640 10 monclient(hunting): authenticate will time out at 862.829455s 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee1357640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fdedc0a5fc0 0x7fdedc0a63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:39896/0 (socket says 172.21.15.121:39896) 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee1357640 1 -- 172.21.15.121:0/670918824 learned_addr learned my addr 172.21.15.121:0/670918824 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.199+0000 7fdee1b58640 1 --1- 172.21.15.121:0/670918824 >> v1:172.21.15.121:6792/0 conn(0x7fdedc09dcf0 0x7fdedc141900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44224/0 (socket says 172.21.15.121:44224) 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.201+0000 7fdee1357640 1 -- 172.21.15.121:0/670918824 >> v1:172.21.15.121:6791/0 conn(0x7fdedc0a5fc0 legacy=0x7fdedc0a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.201+0000 7fdee1b58640 1 -- 172.21.15.121:0/670918824 >> v1:172.21.15.121:6792/0 conn(0x7fdedc09dcf0 legacy=0x7fdedc141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdee0b56640 1 -- 172.21.15.121:0/670918824 >> v1:172.21.15.153:6790/0 conn(0x7fdedc0a6ab0 legacy=0x7fdedc13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2949230001 0 0) 0x7fdedc09d310 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): my global_id is 4621 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdec4003520 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3635317366 0 0) 0x7fdedc09e180 con 0x7fdedc09dcf0 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient(hunting): my global_id is 4722 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc09d310 con 0x7fdedc09dcf0 2024-09-13T13:53:03.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 974221432 0 0) 0x7fdec4003520 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 >> v1:172.21.15.121:6792/0 conn(0x7fdedc09dcf0 legacy=0x7fdedc141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 >> v1:172.21.15.153:6790/0 conn(0x7fdedc0a6ab0 legacy=0x7fdedc13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 monclient: found mon.noname-c 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc0a6ed0 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdecc001e60 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdee35bb640 5 monclient: authenticate success, global_id 4621 2024-09-13T13:53:03.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdee35bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: dump: 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.214 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdee35bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdecc004c20 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 1 -- 172.21.15.121:0/670918824 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdecc005190 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdedbfff640 10 monclient: dump: 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.216 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.217 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdee35bb640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.202+0000 7fdee35bb640 4 set_mon_vals no callback set 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 10 monclient: shutdown 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 1 -- 172.21.15.121:0/670918824 >> v1:172.21.15.121:6791/0 conn(0x7fdedc0a5fc0 legacy=0x7fdedc0a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 1 -- 172.21.15.121:0/670918824 shutdown_connections 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 1 -- 172.21.15.121:0/670918824 >> 172.21.15.121:0/670918824 conn(0x7fdedc0ab9e0 msgr2=0x7fdedc0abdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 1 -- 172.21.15.121:0/670918824 shutdown_connections 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 1 -- 172.21.15.121:0/670918824 wait complete. 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.203+0000 7fdee35bb640 10 monclient: monmap: 2024-09-13T13:53:03.218 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.219 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 1 Processor -- start 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 1 -- start start 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient: init 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient: _add_conns ranks=[1,3,5,6,0,8,2,7,4] 2024-09-13T13:53:03.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient(hunting): picked mon.noname-f con 0x7fdedc09dcf0 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient(hunting): picked mon.noname-g con 0x7fdedc0a5fc0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient(hunting): picked mon.noname-h con 0x7fdedc0a6ab0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc129390 con 0x7fdedc09dcf0 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc126390 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc0a8e10 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee35bb640 10 monclient(hunting): authenticate will time out at 862.834576s 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee0b56640 1 --1- >> v1:172.21.15.153:6790/0 conn(0x7fdedc0a5fc0 0x7fdedc125c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6790/0 says I am v1:172.21.15.121:38808/0 (socket says 172.21.15.121:38808) 2024-09-13T13:53:03.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.204+0000 7fdee0b56640 1 -- 172.21.15.121:0/997135277 learned_addr learned my addr 172.21.15.121:0/997135277 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fdee0b56640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6790/0 conn(0x7fdedc0a5fc0 legacy=0x7fdedc125c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fdee1b58640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6791/0 conn(0x7fdedc0a6ab0 legacy=0x7fdedc1d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fdee1357640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6789/0 conn(0x7fdedc09dcf0 legacy=0x7fdedc125570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606901828 0 0) 0x7fdedc126390 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): my global_id is 4584 2024-09-13T13:53:03.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdec8003550 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532591451 0 0) 0x7fdedc129390 con 0x7fdedc09dcf0 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): my global_id is 104212 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc126390 con 0x7fdedc09dcf0 2024-09-13T13:53:03.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183404422 0 0) 0x7fdedc0a8e10 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): my global_id is 4631 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdedc129390 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2167095727 0 0) 0x7fdedc126390 con 0x7fdedc09dcf0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdedc0a8e10 con 0x7fdedc09dcf0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 379807111 0 0) 0x7fdedc129390 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdedc126390 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdecc005110 con 0x7fdedc09dcf0 2024-09-13T13:53:03.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-f (according to old e1) 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.230 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2284660626 0 0) 0x7fdec8003550 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdedc129390 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fded4001320 con 0x7fdedc0a6ab0 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): got monmap 1 from mon.h (according to old e1) 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.232 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fded0001d20 con 0x7fdedc0a5fc0 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.233 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1440720528 0 0) 0x7fdedc0a8e10 con 0x7fdedc09dcf0 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.207+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6791/0 conn(0x7fdedc0a6ab0 legacy=0x7fdedc1d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6790/0 conn(0x7fdedc0a5fc0 legacy=0x7fdedc125c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 1 monclient: found mon.f 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc0a88c0 con 0x7fdedc09dcf0 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 5 monclient: authenticate success, global_id 104212 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 10 monclient: _renew_subs 2024-09-13T13:53:03.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc141ee0 con 0x7fdedc09dcf0 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdecc0016a0 con 0x7fdedc09dcf0 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fdecc001e00 con 0x7fdedc09dcf0 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fded9ffb640 10 monclient: dump: 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.236 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 10 monclient: _renew_subs 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdedc127030 con 0x7fdedc09dcf0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdedb7fe640 4 set_mon_vals no callback set 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 10 monclient: _send_mon_message to mon.f at v1:172.21.15.153:6789/0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.208+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.153:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fdedc0a29c0 con 0x7fdedc09dcf0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fdecc011fa0 con 0x7fdedc09dcf0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fdecc04e540 con 0x7fdedc09dcf0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.121:6808/3060725185 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdecc004c20 con 0x7fdec80473f0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== mon.1 v1:172.21.15.153:6789/0 9 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (unknown 2135480514 0 0) 0x7fdecc04e9b0 con 0x7fdedc09dcf0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fded9ffb640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fdee1b58640 10 monclient: get_auth_request con 0x7fdec80473f0 auth_method 0 2024-09-13T13:53:03.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.209+0000 7fdee0b56640 10 monclient: get_auth_request con 0x7fdec80406c0 auth_method 0 2024-09-13T13:53:03.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.214+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== osd.0 v1:172.21.15.121:6808/3060725185 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdecc004c20 con 0x7fdec80473f0 2024-09-13T13:53:03.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.226+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 --> v1:172.21.15.121:6808/3060725185 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdedc129010 con 0x7fdec80473f0 2024-09-13T13:53:03.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.226+0000 7fded9ffb640 1 -- 172.21.15.121:0/997135277 <== osd.0 v1:172.21.15.121:6808/3060725185 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fdedc129010 con 0x7fdec80473f0 2024-09-13T13:53:03.240 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:53:03.240 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:53:03.241 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.121:6808/3060725185 conn(0x7fdec80473f0 legacy=0x7fdec8049810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6812/1206688214 conn(0x7fdec80406c0 legacy=0x7fdec8042b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 10 monclient: shutdown 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 >> v1:172.21.15.153:6789/0 conn(0x7fdedc09dcf0 legacy=0x7fdedc125570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee1b58640 1 -- 172.21.15.121:0/997135277 reap_dead start 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 shutdown_connections 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 >> 172.21.15.121:0/997135277 conn(0x7fdedc0ab9e0 msgr2=0x7fdedc142890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 shutdown_connections 2024-09-13T13:53:03.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.227+0000 7fdee35bb640 1 -- 172.21.15.121:0/997135277 wait complete. 2024-09-13T13:53:03.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.340+0000 7f502d900640 10 monclient: get_monmap_and_config 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.340+0000 7f502d900640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.340+0000 7f502d900640 10 monclient: monmap: 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:03.341934+0000 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:03.341934+0000 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.343 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.340+0000 7f502d900640 1 Processor -- start 2024-09-13T13:53:03.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.340+0000 7f502d900640 1 -- start start 2024-09-13T13:53:03.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.340+0000 7f502d900640 10 monclient: init 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient: _add_conns ranks=[7,5,8,2,3,6,4,1,0] 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient(hunting): picked mon.noname-i con 0x7f5028138bd0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient(hunting): picked mon.noname-h con 0x7f5028138fb0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient(hunting): picked mon.noname-e con 0x7f5028139420 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f502809e100 con 0x7f5028138fb0 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f502809db40 con 0x7f5028138bd0 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f502809de20 con 0x7f5028139420 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f502d900640 10 monclient(hunting): authenticate will time out at 862.971439s 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f50277fe640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f5028139420 0x7f5028141680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56872/0 (socket says 172.21.15.121:56872) 2024-09-13T13:53:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.341+0000 7f50277fe640 1 -- 172.21.15.121:0/2906446138 learned_addr learned my addr 172.21.15.121:0/2906446138 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.343+0000 7f50277fe640 1 -- 172.21.15.121:0/2906446138 >> v1:172.21.15.121:6793/0 conn(0x7f5028139420 legacy=0x7f5028141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.343+0000 7f5026ffd640 1 -- 172.21.15.121:0/2906446138 >> v1:172.21.15.153:6792/0 conn(0x7f5028138bd0 legacy=0x7f50280a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.343+0000 7f50267fc640 1 -- 172.21.15.121:0/2906446138 >> v1:172.21.15.153:6791/0 conn(0x7f5028138fb0 legacy=0x7f502813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.343+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3815752260 0 0) 0x7f502809de20 con 0x7f5028139420 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): my global_id is 4652 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5010003520 con 0x7f5028139420 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1293795473 0 0) 0x7f502809e100 con 0x7f5028138fb0 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): my global_id is 4640 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502809de20 con 0x7f5028138fb0 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085123374 0 0) 0x7f502809db40 con 0x7f5028138bd0 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient(hunting): my global_id is 104209 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502809e100 con 0x7f5028138bd0 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 188000554 0 0) 0x7f5010003520 con 0x7f5028139420 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 >> v1:172.21.15.153:6792/0 conn(0x7f5028138bd0 legacy=0x7f50280a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 >> v1:172.21.15.153:6791/0 conn(0x7f5028138fb0 legacy=0x7f502813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 monclient: found mon.noname-e 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50280a5da0 con 0x7f5028139420 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 5 monclient: authenticate success, global_id 4652 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5020001f20 con 0x7f5028139420 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: dump: 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.356 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5020004c20 con 0x7f5028139420 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 1 -- 172.21.15.121:0/2906446138 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f5020005150 con 0x7f5028139420 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 4 set_mon_vals no callback set 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f5025ffb640 10 monclient: dump: 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.358 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 10 monclient: shutdown 2024-09-13T13:53:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 1 -- 172.21.15.121:0/2906446138 >> v1:172.21.15.121:6793/0 conn(0x7f5028139420 legacy=0x7f5028141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 1 -- 172.21.15.121:0/2906446138 shutdown_connections 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 1 -- 172.21.15.121:0/2906446138 >> 172.21.15.121:0/2906446138 conn(0x7f50280ab800 msgr2=0x7f50280abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 1 -- 172.21.15.121:0/2906446138 shutdown_connections 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.344+0000 7f502d900640 1 -- 172.21.15.121:0/2906446138 wait complete. 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.345+0000 7f502d900640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.345+0000 7f502d900640 10 monclient: monmap: 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.345+0000 7f502d900640 1 Processor -- start 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.345+0000 7f502d900640 1 -- start start 2024-09-13T13:53:03.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.345+0000 7f502d900640 10 monclient: init 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient: _add_conns ranks=[5,0,7,1,3,4,6,2,8] 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient(hunting): picked mon.noname-h con 0x7f5028138bd0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient(hunting): picked mon.noname-a con 0x7f5028138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient(hunting): picked mon.noname-i con 0x7f5028139420 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50280a77e0 con 0x7f5028138fb0 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f502813b370 con 0x7f5028138bd0 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50281ced50 con 0x7f5028139420 2024-09-13T13:53:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f502d900640 10 monclient(hunting): authenticate will time out at 862.976474s 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f50267fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5028138fb0 0x7f50281ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51586/0 (socket says 172.21.15.121:51586) 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.346+0000 7f50267fc640 1 -- 172.21.15.121:0/3127412335 learned_addr learned my addr 172.21.15.121:0/3127412335 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f50267fc640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.121:6789/0 conn(0x7f5028138fb0 legacy=0x7f50281ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2063618973 0 0) 0x7f502809de20 con 0x7f5028138fb0 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f50277fe640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.153:6792/0 conn(0x7f5028139420 legacy=0x7f50281d6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 10 monclient(hunting): my global_id is 4707 2024-09-13T13:53:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f5026ffd640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.153:6791/0 conn(0x7f5028138bd0 legacy=0x7f50281cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5014003520 con 0x7f5028138fb0 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1136429833 0 0) 0x7f5014003520 con 0x7f5028138fb0 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.348+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f502809de20 con 0x7f5028138fb0 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f501c003610 con 0x7f5028138fb0 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): dump: 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017153974 0 0) 0x7f502813b370 con 0x7f5028138bd0 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): my global_id is 4649 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5014003520 con 0x7f5028138bd0 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025605187 0 0) 0x7f50281ced50 con 0x7f5028139420 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient(hunting): my global_id is 104218 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f502813b370 con 0x7f5028139420 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4050809335 0 0) 0x7f502809de20 con 0x7f5028138fb0 2024-09-13T13:53:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.153:6792/0 conn(0x7f5028139420 legacy=0x7f50281d6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.153:6791/0 conn(0x7f5028138bd0 legacy=0x7f50281cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 monclient: found mon.a 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50280a7d30 con 0x7f5028138fb0 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 5 monclient: authenticate success, global_id 4707 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 10 monclient: _renew_subs 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5028139e50 con 0x7f5028138fb0 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 10 monclient: _renew_subs 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:03.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50281425b0 con 0x7f5028138fb0 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f501c003bd0 con 0x7f5028138fb0 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f501c004100 con 0x7f5028138fb0 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f50257fa640 4 set_mon_vals no callback set 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.349+0000 7f500ffff640 10 monclient: dump: 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f502d900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f501c0042a0 con 0x7f5028138fb0 2024-09-13T13:53:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f502d900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f502d900640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.121:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5008005080 con 0x7f5028138fb0 2024-09-13T13:53:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f501c04e2d0 con 0x7f5028138fb0 2024-09-13T13:53:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.350+0000 7f5026ffd640 10 monclient: get_auth_request con 0x7f501403fd40 auth_method 0 2024-09-13T13:53:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.354+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f50280a77e0 con 0x7f5028138fb0 2024-09-13T13:53:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.354+0000 7f500ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.354+0000 7f500ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:03.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.536+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f5008000b90 con 0x7f501403fd40 2024-09-13T13:53:03.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.537+0000 7f500ffff640 1 -- 172.21.15.121:0/3127412335 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f5008000b90 con 0x7f501403fd40 2024-09-13T13:53:03.539 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T13:53:03.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.153:6812/1206688214 conn(0x7f501403fd40 legacy=0x7f5014042200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 10 monclient: shutdown 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 >> v1:172.21.15.121:6789/0 conn(0x7f5028138fb0 legacy=0x7f50281ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 shutdown_connections 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 >> 172.21.15.121:0/3127412335 conn(0x7f50280ab800 msgr2=0x7f5028137af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 shutdown_connections 2024-09-13T13:53:03.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.539+0000 7f502d900640 1 -- 172.21.15.121:0/3127412335 wait complete. 2024-09-13T13:53:03.551 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T13:53:03.552 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:53:03.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.652+0000 7fbf6a6f7640 10 monclient: get_monmap_and_config 2024-09-13T13:53:03.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.652+0000 7fbf6a6f7640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.652+0000 7fbf6a6f7640 10 monclient: monmap: 2024-09-13T13:53:03.654 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.654 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:03.654211+0000 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:03.654211+0000 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.655 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 1 Processor -- start 2024-09-13T13:53:03.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 1 -- start start 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient: init 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient: _add_conns ranks=[4,2,6,8,3,7,0,5,1] 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient(hunting): picked mon.noname-c con 0x7fbf640a5fc0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient(hunting): picked mon.noname-b con 0x7fbf640a6ab0 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient(hunting): picked mon.noname-d con 0x7fbf6409ddd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf640a71f0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf640a7410 con 0x7fbf640a5fc0 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf6406f840 con 0x7fbf6409ddd0 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf6a6f7640 10 monclient(hunting): authenticate will time out at 863.283814s 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf637fe640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fbf640a6ab0 0x7fbf6413e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:48984/0 (socket says 172.21.15.121:48984) 2024-09-13T13:53:03.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf68c94640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fbf6409ddd0 0x7fbf64141900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44228/0 (socket says 172.21.15.121:44228) 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.653+0000 7fbf637fe640 1 -- 172.21.15.121:0/504805451 learned_addr learned my addr 172.21.15.121:0/504805451 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf637fe640 1 -- 172.21.15.121:0/504805451 >> v1:172.21.15.121:6790/0 conn(0x7fbf640a6ab0 legacy=0x7fbf6413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf68c94640 1 -- 172.21.15.121:0/504805451 >> v1:172.21.15.121:6792/0 conn(0x7fbf6409ddd0 legacy=0x7fbf64141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf63fff640 1 -- 172.21.15.121:0/504805451 >> v1:172.21.15.121:6791/0 conn(0x7fbf640a5fc0 legacy=0x7fbf640a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 236559224 0 0) 0x7fbf640a71f0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): my global_id is 4646 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf4c003520 con 0x7fbf640a6ab0 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1075383817 0 0) 0x7fbf6406f840 con 0x7fbf6409ddd0 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): my global_id is 4731 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf640a71f0 con 0x7fbf6409ddd0 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208742551 0 0) 0x7fbf640a7410 con 0x7fbf640a5fc0 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient(hunting): my global_id is 4630 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf6406f840 con 0x7fbf640a5fc0 2024-09-13T13:53:03.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 134843278 0 0) 0x7fbf4c003520 con 0x7fbf640a6ab0 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 >> v1:172.21.15.121:6792/0 conn(0x7fbf6409ddd0 legacy=0x7fbf64141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 >> v1:172.21.15.121:6791/0 conn(0x7fbf640a5fc0 legacy=0x7fbf640a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 monclient: found mon.noname-b 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf640a7720 con 0x7fbf640a6ab0 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbf58001ae0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf6a6f7640 5 monclient: authenticate success, global_id 4646 2024-09-13T13:53:03.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf6a6f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: dump: 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.665 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf6a6f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf580048a0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 1 -- 172.21.15.121:0/504805451 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbf58004d90 con 0x7fbf640a6ab0 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf6a6f7640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf6a6f7640 4 set_mon_vals no callback set 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.656+0000 7fbf62ffd640 10 monclient: dump: 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.667 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 10 monclient: shutdown 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/504805451 >> v1:172.21.15.121:6790/0 conn(0x7fbf640a6ab0 legacy=0x7fbf6413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/504805451 shutdown_connections 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/504805451 >> 172.21.15.121:0/504805451 conn(0x7fbf6406eab0 msgr2=0x7fbf6406eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/504805451 shutdown_connections 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/504805451 wait complete. 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.657+0000 7fbf6a6f7640 10 monclient: monmap: 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.668 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.669 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 1 Processor -- start 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 1 -- start start 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient: init 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient: _add_conns ranks=[6,3,2,4,5,0,1,8,7] 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient(hunting): picked mon.noname-d con 0x7fbf6409ddd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient(hunting): picked mon.noname-g con 0x7fbf640a5fc0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient(hunting): picked mon.noname-b con 0x7fbf640a6ab0 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf64126ec0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf64128980 con 0x7fbf640a5fc0 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf64128c60 con 0x7fbf6409ddd0 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf6a6f7640 10 monclient(hunting): authenticate will time out at 863.288735s 2024-09-13T13:53:03.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf63fff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fbf6409ddd0 0x7fbf641cdad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44244/0 (socket says 172.21.15.121:44244) 2024-09-13T13:53:03.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.658+0000 7fbf63fff640 1 -- 172.21.15.121:0/4064350832 learned_addr learned my addr 172.21.15.121:0/4064350832 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.660+0000 7fbf68c94640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.121:6790/0 conn(0x7fbf640a6ab0 legacy=0x7fbf641d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.660+0000 7fbf63fff640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.121:6792/0 conn(0x7fbf6409ddd0 legacy=0x7fbf641cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184885764 0 0) 0x7fbf64126ec0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): my global_id is 4655 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf50003550 con 0x7fbf640a6ab0 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251077594 0 0) 0x7fbf64128c60 con 0x7fbf6409ddd0 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): my global_id is 4740 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf64126ec0 con 0x7fbf6409ddd0 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf637fe640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.153:6790/0 conn(0x7fbf640a5fc0 legacy=0x7fbf641418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4194937146 0 0) 0x7fbf50003550 con 0x7fbf640a6ab0 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf64128c60 con 0x7fbf640a6ab0 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbf5c004b60 con 0x7fbf640a6ab0 2024-09-13T13:53:03.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): dump: 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.676 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3113562714 0 0) 0x7fbf64126ec0 con 0x7fbf6409ddd0 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf50003550 con 0x7fbf6409ddd0 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbf54003610 con 0x7fbf6409ddd0 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): dump: 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.677 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286958361 0 0) 0x7fbf64128980 con 0x7fbf640a5fc0 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient(hunting): my global_id is 4593 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf64126ec0 con 0x7fbf640a5fc0 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 29175062 0 0) 0x7fbf64128c60 con 0x7fbf640a6ab0 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.121:6792/0 conn(0x7fbf6409ddd0 legacy=0x7fbf641cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.153:6790/0 conn(0x7fbf640a5fc0 legacy=0x7fbf641418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 monclient: found mon.b 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf64141fd0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 5 monclient: authenticate success, global_id 4655 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 10 monclient: _renew_subs 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf64127a20 con 0x7fbf640a6ab0 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 10 monclient: _renew_subs 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:03.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf641d70d0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf5c004fb0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fbf5c0054a0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf627fc640 4 set_mon_vals no callback set 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.661+0000 7fbf60ff9640 10 monclient: dump: 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.681 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fbf5c011ba0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf6a6f7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fbf44000f80 con 0x7fbf640a6ab0 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fbf5c04e4a0 con 0x7fbf640a6ab0 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6800/1651307734 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf5004aaf0 con 0x7fbf50046e80 2024-09-13T13:53:03.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (unknown 2135480514 0 0) 0x7fbf5c017240 con 0x7fbf640a6ab0 2024-09-13T13:53:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf60ff9640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-09-13T13:53:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf63fff640 10 monclient: get_auth_request con 0x7fbf50040230 auth_method 0 2024-09-13T13:53:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.662+0000 7fbf637fe640 10 monclient: get_auth_request con 0x7fbf50046e80 auth_method 0 2024-09-13T13:53:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.667+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== osd.1 v1:172.21.15.121:6800/1651307734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fbf5004aaf0 con 0x7fbf50046e80 2024-09-13T13:53:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 --> v1:172.21.15.121:6800/1651307734 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbf44002cd0 con 0x7fbf50046e80 2024-09-13T13:53:03.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf60ff9640 1 -- 172.21.15.121:0/4064350832 <== osd.1 v1:172.21.15.121:6800/1651307734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fbf44002cd0 con 0x7fbf50046e80 2024-09-13T13:53:03.684 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:53:03.684 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:53:03.684 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:53:03.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.121:6800/1651307734 conn(0x7fbf50046e80 legacy=0x7fbf500492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.153:6812/1206688214 conn(0x7fbf50040230 legacy=0x7fbf500426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 10 monclient: shutdown 2024-09-13T13:53:03.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 >> v1:172.21.15.121:6790/0 conn(0x7fbf640a6ab0 legacy=0x7fbf641d59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf68c94640 1 -- 172.21.15.121:0/4064350832 reap_dead start 2024-09-13T13:53:03.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 shutdown_connections 2024-09-13T13:53:03.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 >> 172.21.15.121:0/4064350832 conn(0x7fbf6406eab0 msgr2=0x7fbf640a4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 shutdown_connections 2024-09-13T13:53:03.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.680+0000 7fbf6a6f7640 1 -- 172.21.15.121:0/4064350832 wait complete. 2024-09-13T13:53:03.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T13:53:03.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.788+0000 7fcb6d03c640 10 monclient: get_monmap_and_config 2024-09-13T13:53:03.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.788+0000 7fcb6d03c640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.788+0000 7fcb6d03c640 10 monclient: monmap: 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:03.790574+0000 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:03.790574+0000 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.791 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.792 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.792 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.792 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.792 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 1 Processor -- start 2024-09-13T13:53:03.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 1 -- start start 2024-09-13T13:53:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient: init 2024-09-13T13:53:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient: _add_conns ranks=[6,0,3,4,8,7,1,2,5] 2024-09-13T13:53:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient(hunting): picked mon.noname-d con 0x7fcb68138bd0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient(hunting): picked mon.noname-a con 0x7fcb68138fb0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:03.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient(hunting): picked mon.noname-g con 0x7fcb68139420 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6809e100 con 0x7fcb68138fb0 2024-09-13T13:53:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6809db40 con 0x7fcb68139420 2024-09-13T13:53:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6809de20 con 0x7fcb68138bd0 2024-09-13T13:53:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.789+0000 7fcb6d03c640 10 monclient(hunting): authenticate will time out at 863.419987s 2024-09-13T13:53:03.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.790+0000 7fcb5e59c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcb68138fb0 0x7fcb6813df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51588/0 (socket says 172.21.15.121:51588) 2024-09-13T13:53:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.790+0000 7fcb5e59c640 1 -- 172.21.15.121:0/1012398410 learned_addr learned my addr 172.21.15.121:0/1012398410 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.790+0000 7fcb66d9d640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fcb68138bd0 0x7fcb680a5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44246/0 (socket says 172.21.15.121:44246) 2024-09-13T13:53:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb66d9d640 1 -- 172.21.15.121:0/1012398410 >> v1:172.21.15.121:6792/0 conn(0x7fcb68138bd0 legacy=0x7fcb680a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb5e59c640 1 -- 172.21.15.121:0/1012398410 >> v1:172.21.15.121:6789/0 conn(0x7fcb68138fb0 legacy=0x7fcb6813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2342481226 0 0) 0x7fcb6809de20 con 0x7fcb68138bd0 2024-09-13T13:53:03.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): my global_id is 4749 2024-09-13T13:53:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb54003520 con 0x7fcb68138bd0 2024-09-13T13:53:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603289017 0 0) 0x7fcb6809e100 con 0x7fcb68138fb0 2024-09-13T13:53:03.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): my global_id is 4716 2024-09-13T13:53:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb6809de20 con 0x7fcb68138fb0 2024-09-13T13:53:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6759e640 1 -- 172.21.15.121:0/1012398410 >> v1:172.21.15.153:6790/0 conn(0x7fcb68139420 legacy=0x7fcb68141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3107384061 0 0) 0x7fcb6809db40 con 0x7fcb68139420 2024-09-13T13:53:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient(hunting): my global_id is 4602 2024-09-13T13:53:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb6809e100 con 0x7fcb68139420 2024-09-13T13:53:03.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4264512469 0 0) 0x7fcb54003520 con 0x7fcb68138bd0 2024-09-13T13:53:03.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 >> v1:172.21.15.153:6790/0 conn(0x7fcb68139420 legacy=0x7fcb68141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 >> v1:172.21.15.121:6789/0 conn(0x7fcb68138fb0 legacy=0x7fcb6813df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 monclient: found mon.noname-d 2024-09-13T13:53:03.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:53:03.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb680a5da0 con 0x7fcb68138bd0 2024-09-13T13:53:03.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcb58001ae0 con 0x7fcb68138bd0 2024-09-13T13:53:03.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6d03c640 5 monclient: authenticate success, global_id 4749 2024-09-13T13:53:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6d03c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: dump: 2024-09-13T13:53:03.814 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.815 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.816 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.817 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.817 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6d03c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:03.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb58001d30 con 0x7fcb68138bd0 2024-09-13T13:53:03.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 1 -- 172.21.15.121:0/1012398410 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcb58004b60 con 0x7fcb68138bd0 2024-09-13T13:53:03.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:03.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6659c640 10 monclient: dump: 2024-09-13T13:53:03.818 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.818 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.819 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.819 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.819 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.819 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.819 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.820 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.820 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.820 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.820 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.820 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.820 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.821 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.821 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.821 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6d03c640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:03.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.792+0000 7fcb6d03c640 4 set_mon_vals no callback set 2024-09-13T13:53:03.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 10 monclient: shutdown 2024-09-13T13:53:03.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 1 -- 172.21.15.121:0/1012398410 >> v1:172.21.15.121:6792/0 conn(0x7fcb68138bd0 legacy=0x7fcb680a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 1 -- 172.21.15.121:0/1012398410 shutdown_connections 2024-09-13T13:53:03.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 1 -- 172.21.15.121:0/1012398410 >> 172.21.15.121:0/1012398410 conn(0x7fcb680ab800 msgr2=0x7fcb680abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 1 -- 172.21.15.121:0/1012398410 shutdown_connections 2024-09-13T13:53:03.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.793+0000 7fcb6d03c640 1 -- 172.21.15.121:0/1012398410 wait complete. 2024-09-13T13:53:03.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.794+0000 7fcb6d03c640 10 monclient: build_initial_monmap 2024-09-13T13:53:03.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.794+0000 7fcb6d03c640 10 monclient: monmap: 2024-09-13T13:53:03.823 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:03.824 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:03.824 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:03.824 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:03.824 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:03.824 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:03.824 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:03.826 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:03.826 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:03.826 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:03.826 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:03.826 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:03.826 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.794+0000 7fcb6d03c640 1 Processor -- start 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.794+0000 7fcb6d03c640 1 -- start start 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.794+0000 7fcb6d03c640 10 monclient: init 2024-09-13T13:53:03.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient: _add_conns ranks=[8,4,5,7,6,1,2,0,3] 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient(hunting): picked mon.noname-e con 0x7fcb68138bd0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient(hunting): picked mon.noname-c con 0x7fcb68138fb0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient(hunting): picked mon.noname-h con 0x7fcb68139420 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb680a7ab0 con 0x7fcb68138fb0 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6812e840 con 0x7fcb68139420 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb68130e80 con 0x7fcb68138bd0 2024-09-13T13:53:03.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb6d03c640 10 monclient(hunting): authenticate will time out at 863.425150s 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb66d9d640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fcb68138bd0 0x7fcb6812da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56878/0 (socket says 172.21.15.121:56878) 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.795+0000 7fcb66d9d640 1 -- 172.21.15.121:0/4198159080 learned_addr learned my addr 172.21.15.121:0/4198159080 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5e59c640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.121:6791/0 conn(0x7fcb68138fb0 legacy=0x7fcb6812e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb66d9d640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.121:6793/0 conn(0x7fcb68138bd0 legacy=0x7fcb6812da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204539474 0 0) 0x7fcb680a7ab0 con 0x7fcb68138fb0 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): my global_id is 4639 2024-09-13T13:53:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb6759e640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.153:6791/0 conn(0x7fcb68139420 legacy=0x7fcb681d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb48003520 con 0x7fcb68138fb0 2024-09-13T13:53:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2545127741 0 0) 0x7fcb68130e80 con 0x7fcb68138bd0 2024-09-13T13:53:03.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): my global_id is 4661 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb680a7ab0 con 0x7fcb68138bd0 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496065848 0 0) 0x7fcb6812e840 con 0x7fcb68139420 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): my global_id is 4658 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb68130e80 con 0x7fcb68139420 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1206960716 0 0) 0x7fcb48003520 con 0x7fcb68138fb0 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb6812e840 con 0x7fcb68138fb0 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcb50001ce0 con 0x7fcb68138fb0 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): dump: 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.834 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4199461968 0 0) 0x7fcb680a7ab0 con 0x7fcb68138bd0 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcb48003520 con 0x7fcb68138bd0 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcb58001a60 con 0x7fcb68138bd0 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:03.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient(hunting): dump: 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.836 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3065655835 0 0) 0x7fcb6812e840 con 0x7fcb68138fb0 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.121:6793/0 conn(0x7fcb68138bd0 legacy=0x7fcb6812da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.153:6791/0 conn(0x7fcb68139420 legacy=0x7fcb681d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 monclient: found mon.c 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb681301f0 con 0x7fcb68138fb0 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient: _finish_auth 0 2024-09-13T13:53:03.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb6d03c640 5 monclient: authenticate success, global_id 4639 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb6d03c640 10 monclient: _renew_subs 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb6d03c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.797+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcb68132d10 con 0x7fcb68138fb0 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb6d03c640 10 monclient: _renew_subs 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb6d03c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7fcb681d7880 con 0x7fcb68138fb0 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcb50004a90 con 0x7fcb68138fb0 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fcb50004f80 con 0x7fcb68138fb0 2024-09-13T13:53:03.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb65d9b640 4 set_mon_vals no callback set 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 10 monclient: dump: 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:03.839 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fcb500051e0 con 0x7fcb68138fb0 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.798+0000 7fcb6d03c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.799+0000 7fcb6d03c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.799+0000 7fcb6d03c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.799+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb4c005080 con 0x7fcb68138fb0 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.799+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fcb5004e090 con 0x7fcb68138fb0 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.799+0000 7fcb66d9d640 10 monclient: get_auth_request con 0x7fcb4803ff80 auth_method 0 2024-09-13T13:53:03.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.803+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fcb500176a0 con 0x7fcb68138fb0 2024-09-13T13:53:03.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.803+0000 7fcb5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:03.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.803+0000 7fcb5ffff640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:03.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.977+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fcb4c000b90 con 0x7fcb4803ff80 2024-09-13T13:53:03.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.978+0000 7fcb5ffff640 1 -- 172.21.15.121:0/4198159080 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7fcb4c000b90 con 0x7fcb4803ff80 2024-09-13T13:53:03.980 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.153:6812/1206688214 conn(0x7fcb4803ff80 legacy=0x7fcb48042440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 10 monclient: shutdown 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 >> v1:172.21.15.121:6791/0 conn(0x7fcb68138fb0 legacy=0x7fcb6812e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 shutdown_connections 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 >> 172.21.15.121:0/4198159080 conn(0x7fcb680ab800 msgr2=0x7fcb6812c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 shutdown_connections 2024-09-13T13:53:03.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:03.980+0000 7fcb6d03c640 1 -- 172.21.15.121:0/4198159080 wait complete. 2024-09-13T13:53:03.993 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T13:53:03.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:53:04.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.093+0000 7f8271223640 10 monclient: get_monmap_and_config 2024-09-13T13:53:04.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.093+0000 7f8271223640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.093+0000 7f8271223640 10 monclient: monmap: 2024-09-13T13:53:04.095 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:04.095413+0000 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:04.095413+0000 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.096 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.097 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.097 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 1 Processor -- start 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 1 -- start start 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient: init 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient: _add_conns ranks=[5,3,7,8,6,2,1,4,0] 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient(hunting): picked mon.noname-h con 0x7f826c0a5fc0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient(hunting): picked mon.noname-g con 0x7f826c0a6ab0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient(hunting): picked mon.noname-i con 0x7f826c09ddd0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826c0a71f0 con 0x7f826c0a6ab0 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826c0a7410 con 0x7f826c0a5fc0 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826c06f840 con 0x7f826c09ddd0 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.094+0000 7f8271223640 10 monclient(hunting): authenticate will time out at 863.724840s 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.095+0000 7f826b59e640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7f826c09ddd0 0x7f826c141900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.121:49886/0 (socket says 172.21.15.121:49886) 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.095+0000 7f826ad9d640 1 --1- >> v1:172.21.15.153:6791/0 conn(0x7f826c0a5fc0 0x7f826c0a63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6791/0 says I am v1:172.21.15.121:47560/0 (socket says 172.21.15.121:47560) 2024-09-13T13:53:04.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.095+0000 7f826b59e640 1 -- 172.21.15.121:0/2395616166 learned_addr learned my addr 172.21.15.121:0/2395616166 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f826b59e640 1 -- 172.21.15.121:0/2395616166 >> v1:172.21.15.153:6792/0 conn(0x7f826c09ddd0 legacy=0x7f826c141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f826a59c640 1 -- 172.21.15.121:0/2395616166 >> v1:172.21.15.153:6790/0 conn(0x7f826c0a6ab0 legacy=0x7f826c13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f826ad9d640 1 -- 172.21.15.121:0/2395616166 >> v1:172.21.15.153:6791/0 conn(0x7f826c0a5fc0 legacy=0x7f826c0a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1957131038 0 0) 0x7f826c0a7410 con 0x7f826c0a5fc0 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): my global_id is 4667 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8258003520 con 0x7f826c0a5fc0 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046525515 0 0) 0x7f826c0a71f0 con 0x7f826c0a6ab0 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): my global_id is 4611 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c0a7410 con 0x7f826c0a6ab0 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2767521039 0 0) 0x7f826c06f840 con 0x7f826c09ddd0 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient(hunting): my global_id is 104227 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c0a71f0 con 0x7f826c09ddd0 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.5 v1:172.21.15.153:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3434001357 0 0) 0x7f8258003520 con 0x7f826c0a5fc0 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 >> v1:172.21.15.153:6792/0 conn(0x7f826c09ddd0 legacy=0x7f826c141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 >> v1:172.21.15.153:6790/0 conn(0x7f826c0a6ab0 legacy=0x7f826c13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 monclient: found mon.noname-h 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: _send_mon_message to mon.noname-h at v1:172.21.15.153:6791/0 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 --> v1:172.21.15.153:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826c0a7720 con 0x7f826c0a5fc0 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8271223640 5 monclient: authenticate success, global_id 4667 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.5 v1:172.21.15.153:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8254001ae0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8271223640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8269d9b640 10 monclient: dump: 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.106 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.097+0000 7f8271223640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.5 v1:172.21.15.153:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f82540047e0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8269d9b640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:04.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 4 set_mon_vals no callback set 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8269d9b640 1 -- 172.21.15.121:0/2395616166 <== mon.5 v1:172.21.15.153:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8254004d90 con 0x7f826c0a5fc0 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8269d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8269d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8269d9b640 10 monclient: dump: 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.108 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 10 monclient: shutdown 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 1 -- 172.21.15.121:0/2395616166 >> v1:172.21.15.153:6791/0 conn(0x7f826c0a5fc0 legacy=0x7f826c0a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 1 -- 172.21.15.121:0/2395616166 shutdown_connections 2024-09-13T13:53:04.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 1 -- 172.21.15.121:0/2395616166 >> 172.21.15.121:0/2395616166 conn(0x7f826c06eab0 msgr2=0x7f826c06eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 1 -- 172.21.15.121:0/2395616166 shutdown_connections 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.098+0000 7f8271223640 1 -- 172.21.15.121:0/2395616166 wait complete. 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.099+0000 7f8271223640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.099+0000 7f8271223640 10 monclient: monmap: 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.110 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.099+0000 7f8271223640 1 Processor -- start 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.099+0000 7f8271223640 1 -- start start 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.099+0000 7f8271223640 10 monclient: init 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient: _add_conns ranks=[1,7,3,6,8,2,0,5,4] 2024-09-13T13:53:04.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient(hunting): picked mon.noname-f con 0x7f826c09ddd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient(hunting): picked mon.noname-i con 0x7f826c0a5fc0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient(hunting): picked mon.noname-g con 0x7f826c0a6ab0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826c128e00 con 0x7f826c09ddd0 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826c1290e0 con 0x7f826c0a6ab0 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826c141e70 con 0x7f826c0a5fc0 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f8271223640 10 monclient(hunting): authenticate will time out at 863.730322s 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f826a59c640 1 --1- >> v1:172.21.15.153:6792/0 conn(0x7f826c0a5fc0 0x7f826c1d2840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6792/0 says I am v1:172.21.15.121:49896/0 (socket says 172.21.15.121:49896) 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f826ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f826c09ddd0 0x7f826c141760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.121:46810/0 (socket says 172.21.15.121:46810) 2024-09-13T13:53:04.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.100+0000 7f826a59c640 1 -- 172.21.15.121:0/1716511563 learned_addr learned my addr 172.21.15.121:0/1716511563 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f826b59e640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6790/0 conn(0x7f826c0a6ab0 legacy=0x7f826c1d5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f826ad9d640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6789/0 conn(0x7f826c09ddd0 legacy=0x7f826c141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f826a59c640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6792/0 conn(0x7f826c0a5fc0 legacy=0x7f826c1d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876884407 0 0) 0x7f826c1290e0 con 0x7f826c0a6ab0 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f82637fe640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f82637fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f82637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.102+0000 7f82637fe640 10 monclient(hunting): my global_id is 4620 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f824c003550 con 0x7f826c0a6ab0 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 61388226 0 0) 0x7f826c141e70 con 0x7f826c0a5fc0 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): my global_id is 104236 2024-09-13T13:53:04.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c1290e0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675428524 0 0) 0x7f826c128e00 con 0x7f826c09ddd0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): my global_id is 104221 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f826c141e70 con 0x7f826c09ddd0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 502720879 0 0) 0x7f826c1290e0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f826c128e00 con 0x7f826c0a5fc0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f825c001d20 con 0x7f826c0a5fc0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): got monmap 1 from mon.noname-i (according to old e1) 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): dump: 2024-09-13T13:53:04.119 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.120 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.3 v1:172.21.15.153:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 496585964 0 0) 0x7f824c003550 con 0x7f826c0a6ab0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f826c1290e0 con 0x7f826c0a6ab0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.1 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3974908821 0 0) 0x7f826c141e70 con 0x7f826c09ddd0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f824c003550 con 0x7f826c09ddd0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.3 v1:172.21.15.153:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8264002080 con 0x7f826c0a6ab0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): got monmap 1 from mon.g (according to old e1) 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): dump: 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.121 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.1 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f8254004cd0 con 0x7f826c09ddd0 2024-09-13T13:53:04.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): got monmap 1 from mon.f (according to old e1) 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient(hunting): dump: 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.123 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2554424805 0 0) 0x7f826c128e00 con 0x7f826c0a5fc0 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6790/0 conn(0x7f826c0a6ab0 legacy=0x7f826c1d5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6789/0 conn(0x7f826c09ddd0 legacy=0x7f826c141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 monclient: found mon.i 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826c0a8b80 con 0x7f826c0a5fc0 2024-09-13T13:53:04.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 5 monclient: authenticate success, global_id 104236 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 10 monclient: _renew_subs 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6792/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f826c126d80 con 0x7f826c0a5fc0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 10 monclient: _renew_subs 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6792/0 -- mon_subscribe({osdmap=0}) -- 0x7f826c128840 con 0x7f826c0a5fc0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f825c0038c0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f825c003eb0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f826959a640 4 set_mon_vals no callback set 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.103+0000 7f82637fe640 10 monclient: dump: 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.126 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f8271223640 10 monclient: _send_mon_message to mon.i at v1:172.21.15.153:6792/0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.153:6792/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f8250000f80 con 0x7f826c0a5fc0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f825c0040d0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f825c04e0e0 con 0x7f826c0a5fc0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.121:6804/2141781943 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f824c04ae90 con 0x7f824c047220 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== mon.7 v1:172.21.15.153:6792/0 9 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (unknown 2135480514 0 0) 0x7f825c04e550 con 0x7f826c0a5fc0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f82637fe640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.104+0000 7f826ad9d640 10 monclient: get_auth_request con 0x7f824c0405a0 auth_method 0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.105+0000 7f826b59e640 10 monclient: get_auth_request con 0x7f824c047220 auth_method 0 2024-09-13T13:53:04.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.109+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== osd.2 v1:172.21.15.121:6804/2141781943 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f824c04ae90 con 0x7f824c047220 2024-09-13T13:53:04.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 --> v1:172.21.15.121:6804/2141781943 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8250002cd0 con 0x7f824c047220 2024-09-13T13:53:04.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f82637fe640 1 -- 172.21.15.121:0/1716511563 <== osd.2 v1:172.21.15.121:6804/2141781943 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f8250002cd0 con 0x7f824c047220 2024-09-13T13:53:04.128 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:53:04.128 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:53:04.129 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:53:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.121:6804/2141781943 conn(0x7f824c047220 legacy=0x7f824c049640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6812/1206688214 conn(0x7f824c0405a0 legacy=0x7f824c042a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 10 monclient: shutdown 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 >> v1:172.21.15.153:6792/0 conn(0x7f826c0a5fc0 legacy=0x7f826c1d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f826b59e640 1 -- 172.21.15.121:0/1716511563 reap_dead start 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 shutdown_connections 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 >> 172.21.15.121:0/1716511563 conn(0x7f826c06eab0 msgr2=0x7f826c0a4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.122+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 shutdown_connections 2024-09-13T13:53:04.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.123+0000 7f8271223640 1 -- 172.21.15.121:0/1716511563 wait complete. 2024-09-13T13:53:04.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T13:53:04.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.231+0000 7fa5dbc59640 10 monclient: get_monmap_and_config 2024-09-13T13:53:04.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.231+0000 7fa5dbc59640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.231+0000 7fa5dbc59640 10 monclient: monmap: 2024-09-13T13:53:04.233 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.233 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.233 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:04.232805+0000 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:04.232805+0000 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.234 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.231+0000 7fa5dbc59640 1 Processor -- start 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.231+0000 7fa5dbc59640 1 -- start start 2024-09-13T13:53:04.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.231+0000 7fa5dbc59640 10 monclient: init 2024-09-13T13:53:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient: _add_conns ranks=[8,0,3,6,2,7,4,1,5] 2024-09-13T13:53:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient(hunting): picked mon.noname-e con 0x7fa5d40a69f0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient(hunting): picked mon.noname-a con 0x7fa5d40a74e0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient(hunting): picked mon.noname-g con 0x7fa5d40a78f0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:04.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d40736d0 con 0x7fa5d40a74e0 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d406fed0 con 0x7fa5d40a78f0 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d4070360 con 0x7fa5d40a69f0 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5dbc59640 10 monclient(hunting): authenticate will time out at 863.862408s 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5d91f4640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa5d40a74e0 0x7fa5d4142720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51594/0 (socket says 172.21.15.121:51594) 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5d99f5640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fa5d40a69f0 0x7fa5d40a6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56880/0 (socket says 172.21.15.121:56880) 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.232+0000 7fa5d91f4640 1 -- 172.21.15.121:0/2940181272 learned_addr learned my addr 172.21.15.121:0/2940181272 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d99f5640 1 -- 172.21.15.121:0/2940181272 >> v1:172.21.15.121:6793/0 conn(0x7fa5d40a69f0 legacy=0x7fa5d40a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d91f4640 1 -- 172.21.15.121:0/2940181272 >> v1:172.21.15.121:6789/0 conn(0x7fa5d40a74e0 legacy=0x7fa5d4142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819823356 0 0) 0x7fa5d4070360 con 0x7fa5d40a69f0 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): my global_id is 4670 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5c0003520 con 0x7fa5d40a69f0 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 988049144 0 0) 0x7fa5d40736d0 con 0x7fa5d40a74e0 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 10 monclient(hunting): my global_id is 4725 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d4070360 con 0x7fa5d40a74e0 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.234+0000 7fa5da1f6640 1 -- 172.21.15.121:0/2940181272 >> v1:172.21.15.153:6790/0 conn(0x7fa5d40a78f0 legacy=0x7fa5d4145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1058899869 0 0) 0x7fa5c0003520 con 0x7fa5d40a69f0 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 >> v1:172.21.15.153:6790/0 conn(0x7fa5d40a78f0 legacy=0x7fa5d4145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 >> v1:172.21.15.121:6789/0 conn(0x7fa5d40a74e0 legacy=0x7fa5d4142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 monclient: found mon.noname-e 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d4074240 con 0x7fa5d40a69f0 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 5 monclient: authenticate success, global_id 4670 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa5cc001e60 con 0x7fa5d40a69f0 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: dump: 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.250 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.251 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5cc004b60 con 0x7fa5d40a69f0 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 1 -- 172.21.15.121:0/2940181272 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa5cc0050d0 con 0x7fa5d40a69f0 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 4 set_mon_vals no callback set 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5d89f3640 10 monclient: dump: 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.252 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 10 monclient: shutdown 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2940181272 >> v1:172.21.15.121:6793/0 conn(0x7fa5d40a69f0 legacy=0x7fa5d40a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2940181272 shutdown_connections 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2940181272 >> 172.21.15.121:0/2940181272 conn(0x7fa5d40abb80 msgr2=0x7fa5d40abf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2940181272 shutdown_connections 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.235+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2940181272 wait complete. 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.236+0000 7fa5dbc59640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.236+0000 7fa5dbc59640 10 monclient: monmap: 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.254 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.236+0000 7fa5dbc59640 1 Processor -- start 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.236+0000 7fa5dbc59640 1 -- start start 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.236+0000 7fa5dbc59640 10 monclient: init 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient: _add_conns ranks=[5,8,7,0,1,2,3,4,6] 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient(hunting): picked mon.noname-h con 0x7fa5d40a69f0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:04.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient(hunting): picked mon.noname-e con 0x7fa5d40a74e0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient(hunting): picked mon.noname-i con 0x7fa5d40a78f0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d40a8a10 con 0x7fa5d40a69f0 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d409fdf0 con 0x7fa5d40a78f0 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d40a2a50 con 0x7fa5d40a74e0 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5dbc59640 10 monclient(hunting): authenticate will time out at 863.867327s 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5d91f4640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7fa5d40a74e0 0x7fa5d409f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56892/0 (socket says 172.21.15.121:56892) 2024-09-13T13:53:04.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.237+0000 7fa5d91f4640 1 -- 172.21.15.121:0/2259422035 learned_addr learned my addr 172.21.15.121:0/2259422035 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5d91f4640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.121:6793/0 conn(0x7fa5d40a74e0 legacy=0x7fa5d409f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5d99f5640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.153:6791/0 conn(0x7fa5d40a69f0 legacy=0x7fa5d409efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904531041 0 0) 0x7fa5d40a2a50 con 0x7fa5d40a74e0 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): my global_id is 4679 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5b8003550 con 0x7fa5d40a74e0 2024-09-13T13:53:04.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5da1f6640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.153:6792/0 conn(0x7fa5d40a78f0 legacy=0x7fa5d41da540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1948004388 0 0) 0x7fa5b8003550 con 0x7fa5d40a74e0 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5d40a2a50 con 0x7fa5d40a74e0 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa5c4001a20 con 0x7fa5d40a74e0 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): dump: 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.258 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169747009 0 0) 0x7fa5d40a8a10 con 0x7fa5d40a69f0 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): my global_id is 4676 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5b8003550 con 0x7fa5d40a69f0 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869882205 0 0) 0x7fa5d409fdf0 con 0x7fa5d40a78f0 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 10 monclient(hunting): my global_id is 104245 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5d40a8a10 con 0x7fa5d40a78f0 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.239+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4084454714 0 0) 0x7fa5d40a2a50 con 0x7fa5d40a74e0 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.153:6792/0 conn(0x7fa5d40a78f0 legacy=0x7fa5d41da540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.153:6791/0 conn(0x7fa5d40a69f0 legacy=0x7fa5d409efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 1 monclient: found mon.e 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d40a2d30 con 0x7fa5d40a74e0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 5 monclient: authenticate success, global_id 4679 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 10 monclient: _renew_subs 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d40a17d0 con 0x7fa5d40a74e0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 10 monclient: _renew_subs 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5d41dbc60 con 0x7fa5d40a74e0 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa5c4004dd0 con 0x7fa5d40a74e0 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa5c40052c0 con 0x7fa5d40a74e0 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5cbfff640 4 set_mon_vals no callback set 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.240+0000 7fa5ca7fc640 10 monclient: dump: 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.262 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fa5c4005460 con 0x7fa5d40a74e0 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5dbc59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5dbc59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5dbc59640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5bc005080 con 0x7fa5d40a74e0 2024-09-13T13:53:04.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fa5c404e3a0 con 0x7fa5d40a74e0 2024-09-13T13:53:04.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.241+0000 7fa5d99f5640 10 monclient: get_auth_request con 0x7fa5b803fcf0 auth_method 0 2024-09-13T13:53:04.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.246+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fa5c4017950 con 0x7fa5d40a74e0 2024-09-13T13:53:04.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.246+0000 7fa5ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:04.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.246+0000 7fa5ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:04.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.424+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fa5bc000b90 con 0x7fa5b803fcf0 2024-09-13T13:53:04.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.424+0000 7fa5ca7fc640 1 -- 172.21.15.121:0/2259422035 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7fa5bc000b90 con 0x7fa5b803fcf0 2024-09-13T13:53:04.426 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T13:53:04.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.153:6812/1206688214 conn(0x7fa5b803fcf0 legacy=0x7fa5b80421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 10 monclient: shutdown 2024-09-13T13:53:04.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 >> v1:172.21.15.121:6793/0 conn(0x7fa5d40a74e0 legacy=0x7fa5d409f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 shutdown_connections 2024-09-13T13:53:04.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 >> 172.21.15.121:0/2259422035 conn(0x7fa5d40abb80 msgr2=0x7fa5d40a9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 shutdown_connections 2024-09-13T13:53:04.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.427+0000 7fa5dbc59640 1 -- 172.21.15.121:0/2259422035 wait complete. 2024-09-13T13:53:04.439 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T13:53:04.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.538+0000 7fa8e69ec640 10 monclient: get_monmap_and_config 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.538+0000 7fa8e69ec640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.538+0000 7fa8e69ec640 10 monclient: monmap: 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:04.540005+0000 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:04.540005+0000 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.540 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.541 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 1 Processor -- start 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 1 -- start start 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient: init 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient: _add_conns ranks=[6,7,0,4,3,5,1,2,8] 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient(hunting): picked mon.noname-d con 0x7fa8e00a5fc0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient(hunting): picked mon.noname-i con 0x7fa8e00a6ab0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e009dcf0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009d030 con 0x7fa8e009dcf0 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009d310 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e009e180 con 0x7fa8e00a6ab0 2024-09-13T13:53:04.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e69ec640 10 monclient(hunting): authenticate will time out at 864.169695s 2024-09-13T13:53:04.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8dffff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fa8e00a5fc0 0x7fa8e00a63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44260/0 (socket says 172.21.15.121:44260) 2024-09-13T13:53:04.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8e4f89640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa8e009dcf0 0x7fa8e0141900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51600/0 (socket says 172.21.15.121:51600) 2024-09-13T13:53:04.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.539+0000 7fa8dffff640 1 -- 172.21.15.121:0/357292112 learned_addr learned my addr 172.21.15.121:0/357292112 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8dffff640 1 -- 172.21.15.121:0/357292112 >> v1:172.21.15.121:6792/0 conn(0x7fa8e00a5fc0 legacy=0x7fa8e00a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8e4f89640 1 -- 172.21.15.121:0/357292112 >> v1:172.21.15.121:6789/0 conn(0x7fa8e009dcf0 legacy=0x7fa8e0141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8df7fe640 1 -- 172.21.15.121:0/357292112 >> v1:172.21.15.153:6792/0 conn(0x7fa8e00a6ab0 legacy=0x7fa8e013e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660264327 0 0) 0x7fa8e009d310 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient(hunting): my global_id is 4758 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8cc003520 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1473861801 0 0) 0x7fa8cc003520 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 >> v1:172.21.15.153:6792/0 conn(0x7fa8e00a6ab0 legacy=0x7fa8e013e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 >> v1:172.21.15.121:6789/0 conn(0x7fa8e009dcf0 legacy=0x7fa8e0141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 monclient: found mon.noname-d 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e00a6ed0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8d0001ae0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8e69ec640 5 monclient: authenticate success, global_id 4758 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8e69ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: dump: 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.549 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8e69ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8d00048a0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 1 -- 172.21.15.121:0/357292112 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8d0004dd0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8e69ec640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8e69ec640 4 set_mon_vals no callback set 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.542+0000 7fa8deffd640 10 monclient: dump: 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.551 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 10 monclient: shutdown 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 1 -- 172.21.15.121:0/357292112 >> v1:172.21.15.121:6792/0 conn(0x7fa8e00a5fc0 legacy=0x7fa8e00a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 1 -- 172.21.15.121:0/357292112 shutdown_connections 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 1 -- 172.21.15.121:0/357292112 >> 172.21.15.121:0/357292112 conn(0x7fa8e00ab9e0 msgr2=0x7fa8e00abdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 1 -- 172.21.15.121:0/357292112 shutdown_connections 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 1 -- 172.21.15.121:0/357292112 wait complete. 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.543+0000 7fa8e69ec640 10 monclient: monmap: 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.553 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 1 Processor -- start 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 1 -- start start 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient: init 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient: _add_conns ranks=[4,8,2,0,5,7,6,1,3] 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e009dcf0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient(hunting): picked mon.noname-e con 0x7fa8e00a5fc0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e00a6ab0 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e0129730 con 0x7fa8e00a6ab0 2024-09-13T13:53:04.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e01cecd0 con 0x7fa8e009dcf0 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e01288c0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8dffff640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7fa8e009dcf0 0x7fa8e01cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:39940/0 (socket says 172.21.15.121:39940) 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8dffff640 1 -- 172.21.15.121:0/249680722 learned_addr learned my addr 172.21.15.121:0/249680722 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.544+0000 7fa8e69ec640 10 monclient(hunting): authenticate will time out at 864.174996s 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.547+0000 7fa8dffff640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6791/0 conn(0x7fa8e009dcf0 legacy=0x7fa8e01cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.547+0000 7fa8e4f89640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6790/0 conn(0x7fa8e00a6ab0 legacy=0x7fa8e01d6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.547+0000 7fa8df7fe640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6793/0 conn(0x7fa8e00a5fc0 legacy=0x7fa8e01ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8e4f89640 1 --1- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6790/0 conn(0x7fa8e00a6ab0 0x7fa8e01d6110 :-1 s=OPENED pgs=178 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7fa8e0129730 auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 947601262 0 0) 0x7fa8e01288c0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): my global_id is 4688 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8c0003520 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980455560 0 0) 0x7fa8e01cecd0 con 0x7fa8e009dcf0 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): my global_id is 4648 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8e01288c0 con 0x7fa8e009dcf0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 932207520 0 0) 0x7fa8c0003520 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8e01cecd0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2989491713 0 0) 0x7fa8e01288c0 con 0x7fa8e009dcf0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa8c0003520 con 0x7fa8e009dcf0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8c8003610 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): dump: 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.616 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.617 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8d0004d90 con 0x7fa8e009dcf0 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.593+0000 7fa8dcff9640 10 monclient(hunting): dump: 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.618 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2488911683 0 0) 0x7fa8e01cecd0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6791/0 conn(0x7fa8e009dcf0 legacy=0x7fa8e01cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6790/0 conn(0x7fa8e00a6ab0 legacy=0x7fa8e01d6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 monclient: found mon.e 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e01273d0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 5 monclient: authenticate success, global_id 4688 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063564668 0 0) 0x7fa8e0129730 con 0x7fa8e00a6ab0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 10 monclient: _renew_subs 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e0129450 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa8c8003aa0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa8c8003fd0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:04.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 10 monclient: dump: 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.622 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 10 monclient: _renew_subs 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa8e0129b50 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8de7fc640 4 set_mon_vals no callback set 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.594+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fa8c8004170 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.595+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fa8c804e0f0 con 0x7fa8e00a5fc0 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.595+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.153:6808/934815702 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8e01414f0 con 0x7fa8e00632f0 2024-09-13T13:53:04.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.595+0000 7fa8dffff640 10 monclient: get_auth_request con 0x7fa8c0040040 auth_method 0 2024-09-13T13:53:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.595+0000 7fa8e4f89640 10 monclient: get_auth_request con 0x7fa8e00632f0 auth_method 0 2024-09-13T13:53:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.600+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== osd.3 v1:172.21.15.153:6808/934815702 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa8e01414f0 con 0x7fa8e00632f0 2024-09-13T13:53:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.612+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 --> v1:172.21.15.153:6808/934815702 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa8e00a19e0 con 0x7fa8e00632f0 2024-09-13T13:53:04.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8dcff9640 1 -- 172.21.15.121:0/249680722 <== osd.3 v1:172.21.15.153:6808/934815702 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fa8e00a19e0 con 0x7fa8e00632f0 2024-09-13T13:53:04.625 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:53:04.625 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:53:04.625 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:53:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.153:6808/934815702 conn(0x7fa8e00632f0 legacy=0x7fa8e00636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.153:6812/1206688214 conn(0x7fa8c0040040 legacy=0x7fa8c0042500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8e69ec640 10 monclient: shutdown 2024-09-13T13:53:04.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8e69ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 >> v1:172.21.15.121:6793/0 conn(0x7fa8e00a5fc0 legacy=0x7fa8e01ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.613+0000 7fa8e4f89640 1 -- 172.21.15.121:0/249680722 reap_dead start 2024-09-13T13:53:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.614+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 shutdown_connections 2024-09-13T13:53:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.614+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 >> 172.21.15.121:0/249680722 conn(0x7fa8e00ab9e0 msgr2=0x7fa8e00a4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.614+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 shutdown_connections 2024-09-13T13:53:04.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.614+0000 7fa8e69ec640 1 -- 172.21.15.121:0/249680722 wait complete. 2024-09-13T13:53:04.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T13:53:04.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.723+0000 7fb0da518640 10 monclient: get_monmap_and_config 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.723+0000 7fb0da518640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.723+0000 7fb0da518640 10 monclient: monmap: 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:04.725681+0000 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:04.725681+0000 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.726 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.727 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.727 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.724+0000 7fb0da518640 1 Processor -- start 2024-09-13T13:53:04.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.724+0000 7fb0da518640 1 -- start start 2024-09-13T13:53:04.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.724+0000 7fb0da518640 10 monclient: init 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient: _add_conns ranks=[6,1,0,2,8,4,7,5,3] 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient(hunting): picked mon.noname-d con 0x7fb0d40a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient(hunting): picked mon.noname-f con 0x7fb0d409d150 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d40a7cf0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d409d970 con 0x7fb0d40a7cf0 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d40a79e0 con 0x7fb0d409d150 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d409e630 con 0x7fb0d40a6160 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0da518640 10 monclient(hunting): authenticate will time out at 864.355245s 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0d3fff640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7fb0d40a6160 0x7fb0d40a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44272/0 (socket says 172.21.15.121:44272) 2024-09-13T13:53:04.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.725+0000 7fb0d3fff640 1 -- 172.21.15.121:0/619247266 learned_addr learned my addr 172.21.15.121:0/619247266 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d8ab5640 1 -- 172.21.15.121:0/619247266 >> v1:172.21.15.121:6789/0 conn(0x7fb0d40a7cf0 legacy=0x7fb0d4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d3fff640 1 -- 172.21.15.121:0/619247266 >> v1:172.21.15.121:6792/0 conn(0x7fb0d40a6160 legacy=0x7fb0d40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 140175793 0 0) 0x7fb0d409e630 con 0x7fb0d40a6160 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): my global_id is 4767 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0c0003520 con 0x7fb0d40a6160 2024-09-13T13:53:04.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728916954 0 0) 0x7fb0d409d970 con 0x7fb0d40a7cf0 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient(hunting): my global_id is 4734 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d409e630 con 0x7fb0d40a7cf0 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d37fe640 1 -- 172.21.15.121:0/619247266 >> v1:172.21.15.153:6789/0 conn(0x7fb0d409d150 legacy=0x7fb0d413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1654675972 0 0) 0x7fb0c0003520 con 0x7fb0d40a6160 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 >> v1:172.21.15.153:6789/0 conn(0x7fb0d409d150 legacy=0x7fb0d413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 >> v1:172.21.15.121:6789/0 conn(0x7fb0d40a7cf0 legacy=0x7fb0d4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 monclient: found mon.noname-d 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient: _send_mon_message to mon.noname-d at v1:172.21.15.121:6792/0 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 --> v1:172.21.15.121:6792/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d409cde0 con 0x7fb0d40a6160 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0c4001ae0 con 0x7fb0d40a6160 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0da518640 5 monclient: authenticate success, global_id 4767 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0da518640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.727+0000 7fb0d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 10 monclient: dump: 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.735 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.736 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 <== mon.6 v1:172.21.15.121:6792/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0c40048a0 con 0x7fb0d40a6160 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 4 set_mon_vals no callback set 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 1 -- 172.21.15.121:0/619247266 <== mon.6 v1:172.21.15.121:6792/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0c4004e10 con 0x7fb0d40a6160 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0d2ffd640 10 monclient: dump: 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.737 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.738 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 10 monclient: shutdown 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 1 -- 172.21.15.121:0/619247266 >> v1:172.21.15.121:6792/0 conn(0x7fb0d40a6160 legacy=0x7fb0d40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 1 -- 172.21.15.121:0/619247266 shutdown_connections 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 1 -- 172.21.15.121:0/619247266 >> 172.21.15.121:0/619247266 conn(0x7fb0d40ab940 msgr2=0x7fb0d40abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 1 -- 172.21.15.121:0/619247266 shutdown_connections 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.728+0000 7fb0da518640 1 -- 172.21.15.121:0/619247266 wait complete. 2024-09-13T13:53:04.739 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.729+0000 7fb0da518640 10 monclient: build_initial_monmap 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.729+0000 7fb0da518640 10 monclient: monmap: 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:04.740 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.729+0000 7fb0da518640 1 Processor -- start 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.729+0000 7fb0da518640 1 -- start start 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.729+0000 7fb0da518640 10 monclient: init 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient: _add_conns ranks=[2,6,3,5,7,0,4,8,1] 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient(hunting): picked mon.noname-b con 0x7fb0d409d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:04.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient(hunting): picked mon.noname-d con 0x7fb0d40a6160 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient(hunting): picked mon.noname-g con 0x7fb0d40a7cf0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d40a45a0 con 0x7fb0d409d150 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d4125d00 con 0x7fb0d40a7cf0 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d4124ac0 con 0x7fb0d40a6160 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0da518640 10 monclient(hunting): authenticate will time out at 864.360137s 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0d3fff640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fb0d409d150 0x7fb0d4139690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:49012/0 (socket says 172.21.15.121:49012) 2024-09-13T13:53:04.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.730+0000 7fb0d3fff640 1 -- 172.21.15.121:0/260821034 learned_addr learned my addr 172.21.15.121:0/260821034 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:04.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d37fe640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.121:6792/0 conn(0x7fb0d40a6160 legacy=0x7fb0d41d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d3fff640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.121:6790/0 conn(0x7fb0d409d150 legacy=0x7fb0d4139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518060858 0 0) 0x7fb0d4124ac0 con 0x7fb0d40a6160 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): my global_id is 4776 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0b4003520 con 0x7fb0d40a6160 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007622365 0 0) 0x7fb0d40a45a0 con 0x7fb0d409d150 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): my global_id is 4673 2024-09-13T13:53:04.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d4124ac0 con 0x7fb0d409d150 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d8ab5640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.153:6790/0 conn(0x7fb0d40a7cf0 legacy=0x7fb0d41d6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456637852 0 0) 0x7fb0d4125d00 con 0x7fb0d40a7cf0 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _negotiate 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): my global_id is 4638 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0d40a45a0 con 0x7fb0d40a7cf0 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 65389242 0 0) 0x7fb0d4124ac0 con 0x7fb0d409d150 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0d4125d00 con 0x7fb0d409d150 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0c4004d90 con 0x7fb0d409d150 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): dump: 2024-09-13T13:53:04.747 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.748 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1239992922 0 0) 0x7fb0b4003520 con 0x7fb0d40a6160 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb0d4124ac0 con 0x7fb0d40a6160 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0bc003610 con 0x7fb0d40a6160 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient(hunting): dump: 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.749 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1459736630 0 0) 0x7fb0d4125d00 con 0x7fb0d409d150 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.121:6792/0 conn(0x7fb0d40a6160 legacy=0x7fb0d41d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.153:6790/0 conn(0x7fb0d40a7cf0 legacy=0x7fb0d41d6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient: _finish_hunting 0 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 monclient: found mon.b 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d4126e20 con 0x7fb0d409d150 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient: _finish_auth 0 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 10 monclient: _check_auth_tickets 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0da518640 5 monclient: authenticate success, global_id 4673 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0da518640 10 monclient: _renew_subs 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0da518640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.732+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d4126580 con 0x7fb0d409d150 2024-09-13T13:53:04.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 10 monclient: _renew_subs 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb0d4125620 con 0x7fb0d409d150 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb0c4001d30 con 0x7fb0d409d150 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fb0c40048a0 con 0x7fb0d409d150 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d27fc640 4 set_mon_vals no callback set 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 10 monclient: dump: 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.752 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:04.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b8005080 con 0x7fb0d409d150 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.733+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fb0c4011c20 con 0x7fb0d409d150 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.734+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fb0c404e200 con 0x7fb0d409d150 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.734+0000 7fb0d37fe640 10 monclient: get_auth_request con 0x7fb0b40403d0 auth_method 0 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.738+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fb0c4017810 con 0x7fb0d409d150 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.738+0000 7fb0d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:04.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.738+0000 7fb0d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:04.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.915+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fb0b8000b90 con 0x7fb0b40403d0 2024-09-13T13:53:04.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.916+0000 7fb0d0ff9640 1 -- 172.21.15.121:0/260821034 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7fb0b8000b90 con 0x7fb0b40403d0 2024-09-13T13:53:04.918 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T13:53:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.153:6812/1206688214 conn(0x7fb0b40403d0 legacy=0x7fb0b4042890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 10 monclient: shutdown 2024-09-13T13:53:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 >> v1:172.21.15.121:6790/0 conn(0x7fb0d409d150 legacy=0x7fb0d4139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 shutdown_connections 2024-09-13T13:53:04.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 >> 172.21.15.121:0/260821034 conn(0x7fb0d40ab940 msgr2=0x7fb0d41c11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:04.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 shutdown_connections 2024-09-13T13:53:04.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:04.918+0000 7fb0da518640 1 -- 172.21.15.121:0/260821034 wait complete. 2024-09-13T13:53:04.931 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T13:53:04.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:53:05.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.031+0000 7faa4c611640 10 monclient: get_monmap_and_config 2024-09-13T13:53:05.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.031+0000 7faa4c611640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.031+0000 7faa4c611640 10 monclient: monmap: 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:05.033539+0000 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:05.033539+0000 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.034 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.035 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.035 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.032+0000 7faa4c611640 1 Processor -- start 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.032+0000 7faa4c611640 1 -- start start 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.032+0000 7faa4c611640 10 monclient: init 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient: _add_conns ranks=[7,8,3,6,0,5,1,2,4] 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient(hunting): picked mon.noname-i con 0x7faa440a5fc0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:05.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient(hunting): picked mon.noname-e con 0x7faa440a6ab0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient(hunting): picked mon.noname-g con 0x7faa4409ddd0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa440a71f0 con 0x7faa4409ddd0 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa440a7410 con 0x7faa440a5fc0 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4406f840 con 0x7faa440a6ab0 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa4c611640 10 monclient(hunting): authenticate will time out at 864.663343s 2024-09-13T13:53:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa49bac640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7faa440a6ab0 0x7faa4413e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56918/0 (socket says 172.21.15.121:56918) 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.033+0000 7faa49bac640 1 -- 172.21.15.121:0/882449153 learned_addr learned my addr 172.21.15.121:0/882449153 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa49bac640 1 -- 172.21.15.121:0/882449153 >> v1:172.21.15.121:6793/0 conn(0x7faa440a6ab0 legacy=0x7faa4413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772330406 0 0) 0x7faa4406f840 con 0x7faa440a6ab0 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient(hunting): my global_id is 4697 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa30003520 con 0x7faa440a6ab0 2024-09-13T13:53:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa4a3ad640 1 -- 172.21.15.121:0/882449153 >> v1:172.21.15.153:6792/0 conn(0x7faa440a5fc0 legacy=0x7faa440a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa4abae640 1 -- 172.21.15.121:0/882449153 >> v1:172.21.15.153:6790/0 conn(0x7faa4409ddd0 legacy=0x7faa44141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3069373393 0 0) 0x7faa30003520 con 0x7faa440a6ab0 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 >> v1:172.21.15.153:6792/0 conn(0x7faa440a5fc0 legacy=0x7faa440a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 >> v1:172.21.15.153:6790/0 conn(0x7faa4409ddd0 legacy=0x7faa44141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 monclient: found mon.noname-e 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa440a7720 con 0x7faa440a6ab0 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa34001ae0 con 0x7faa440a6ab0 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa4c611640 5 monclient: authenticate success, global_id 4697 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa4c611640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa493ab640 10 monclient: dump: 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.053 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.035+0000 7faa4c611640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa340048a0 con 0x7faa440a6ab0 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa493ab640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa493ab640 1 -- 172.21.15.121:0/882449153 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa34004e10 con 0x7faa440a6ab0 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa493ab640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 4 set_mon_vals no callback set 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa493ab640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:05.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa493ab640 10 monclient: dump: 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 10 monclient: shutdown 2024-09-13T13:53:05.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 1 -- 172.21.15.121:0/882449153 >> v1:172.21.15.121:6793/0 conn(0x7faa440a6ab0 legacy=0x7faa4413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 1 -- 172.21.15.121:0/882449153 shutdown_connections 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 1 -- 172.21.15.121:0/882449153 >> 172.21.15.121:0/882449153 conn(0x7faa4406eab0 msgr2=0x7faa4406eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 1 -- 172.21.15.121:0/882449153 shutdown_connections 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.036+0000 7faa4c611640 1 -- 172.21.15.121:0/882449153 wait complete. 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient: monmap: 2024-09-13T13:53:05.057 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.058 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.059 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.059 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.059 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.059 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.060 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.060 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 1 Processor -- start 2024-09-13T13:53:05.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 1 -- start start 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient: init 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient: _add_conns ranks=[4,7,3,8,5,6,2,0,1] 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient(hunting): picked mon.noname-c con 0x7faa4409ddd0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient(hunting): picked mon.noname-i con 0x7faa440a5fc0 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 10 monclient(hunting): picked mon.noname-g con 0x7faa440a6ab0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.037+0000 7faa4c611640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa441299b0 con 0x7faa440a6ab0 2024-09-13T13:53:05.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.038+0000 7faa4c611640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa441292b0 con 0x7faa4409ddd0 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.038+0000 7faa4c611640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa44129590 con 0x7faa440a5fc0 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.038+0000 7faa4c611640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.038+0000 7faa4c611640 10 monclient(hunting): authenticate will time out at 864.668043s 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.038+0000 7faa4a3ad640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7faa4409ddd0 0x7faa441cdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:39950/0 (socket says 172.21.15.121:39950) 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.038+0000 7faa4a3ad640 1 -- 172.21.15.121:0/1755984181 learned_addr learned my addr 172.21.15.121:0/1755984181 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4a3ad640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.121:6791/0 conn(0x7faa4409ddd0 legacy=0x7faa441cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa49bac640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6792/0 conn(0x7faa440a5fc0 legacy=0x7faa441ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4abae640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6790/0 conn(0x7faa440a6ab0 legacy=0x7faa441d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31380161 0 0) 0x7faa441292b0 con 0x7faa4409ddd0 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): my global_id is 4657 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa28003520 con 0x7faa4409ddd0 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 703257449 0 0) 0x7faa44129590 con 0x7faa440a5fc0 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): my global_id is 104263 2024-09-13T13:53:05.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa441292b0 con 0x7faa440a5fc0 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4abae640 1 --1- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6790/0 conn(0x7faa440a6ab0 0x7faa441d6200 :-1 s=OPENED pgs=131 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7faa440a71f0 auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2095531747 0 0) 0x7faa28003520 con 0x7faa4409ddd0 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa44129590 con 0x7faa4409ddd0 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa40003610 con 0x7faa4409ddd0 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): dump: 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.064 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.7 v1:172.21.15.153:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2082337198 0 0) 0x7faa441292b0 con 0x7faa440a5fc0 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.153:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faa28003520 con 0x7faa440a5fc0 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.7 v1:172.21.15.153:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa34001d30 con 0x7faa440a5fc0 2024-09-13T13:53:05.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): got monmap 1 from mon.i (according to old e1) 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient(hunting): dump: 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.066 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1365881181 0 0) 0x7faa44129590 con 0x7faa4409ddd0 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6792/0 conn(0x7faa440a5fc0 legacy=0x7faa441ce6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6790/0 conn(0x7faa440a6ab0 legacy=0x7faa441d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 monclient: found mon.c 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa441cedc0 con 0x7faa4409ddd0 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 5 monclient: authenticate success, global_id 4657 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 10 monclient: _renew_subs 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803515704 0 0) 0x7faa440a71f0 con 0x7faa440a6ab0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faa440a5430 con 0x7faa4409ddd0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 10 monclient: _renew_subs 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7faa44127230 con 0x7faa4409ddd0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faa40003bc0 con 0x7faa4409ddd0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7faa400040f0 con 0x7faa4409ddd0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa48baa640 4 set_mon_vals no callback set 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.040+0000 7faa3affd640 10 monclient: dump: 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.069 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.041+0000 7faa4c611640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.041+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.121:6791/0 -- mon_get_version(what=osdmap handle=1) -- 0x7faa2c000f80 con 0x7faa4409ddd0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.041+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7faa40011ca0 con 0x7faa4409ddd0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.042+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7faa4004e250 con 0x7faa4409ddd0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.042+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.153:6800/1083052486 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa34001d30 con 0x7faa28046dd0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.042+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (unknown 2135480514 0 0) 0x7faa4004e6c0 con 0x7faa4409ddd0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.042+0000 7faa3affd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.042+0000 7faa49bac640 10 monclient: get_auth_request con 0x7faa28040240 auth_method 0 2024-09-13T13:53:05.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.042+0000 7faa4abae640 10 monclient: get_auth_request con 0x7faa28046dd0 auth_method 0 2024-09-13T13:53:05.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.047+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== osd.4 v1:172.21.15.153:6800/1083052486 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7faa34001d30 con 0x7faa28046dd0 2024-09-13T13:53:05.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.060+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 --> v1:172.21.15.153:6800/1083052486 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faa2c002cd0 con 0x7faa28046dd0 2024-09-13T13:53:05.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.060+0000 7faa3affd640 1 -- 172.21.15.121:0/1755984181 <== osd.4 v1:172.21.15.153:6800/1083052486 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7faa2c002cd0 con 0x7faa28046dd0 2024-09-13T13:53:05.072 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:53:05.072 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:53:05.072 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:53:05.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6800/1083052486 conn(0x7faa28046dd0 legacy=0x7faa280491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.153:6812/1206688214 conn(0x7faa28040240 legacy=0x7faa28042700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 10 monclient: shutdown 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 >> v1:172.21.15.121:6791/0 conn(0x7faa4409ddd0 legacy=0x7faa441cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4abae640 1 -- 172.21.15.121:0/1755984181 reap_dead start 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 shutdown_connections 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 >> 172.21.15.121:0/1755984181 conn(0x7faa4406eab0 msgr2=0x7faa441426d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 shutdown_connections 2024-09-13T13:53:05.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.061+0000 7faa4c611640 1 -- 172.21.15.121:0/1755984181 wait complete. 2024-09-13T13:53:05.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.171+0000 7f12cb389640 10 monclient: get_monmap_and_config 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.171+0000 7f12cb389640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.171+0000 7f12cb389640 10 monclient: monmap: 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:05.172902+0000 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:05.172902+0000 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.173 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.174 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.171+0000 7f12cb389640 1 Processor -- start 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.171+0000 7f12cb389640 1 -- start start 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.171+0000 7f12cb389640 10 monclient: init 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient: _add_conns ranks=[8,2,5,6,3,4,1,7,0] 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient(hunting): picked mon.noname-e con 0x7f12c40a6160 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient(hunting): picked mon.noname-b con 0x7f12c409d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient(hunting): picked mon.noname-h con 0x7f12c40a6c50 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:05.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c409e210 con 0x7f12c409d150 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c40a7ef0 con 0x7f12c40a6c50 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c409d970 con 0x7f12c40a6160 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12cb389640 10 monclient(hunting): authenticate will time out at 864.802466s 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12c8924640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f12c409d150 0x7f12c413e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:49018/0 (socket says 172.21.15.121:49018) 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12c9125640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f12c40a6160 0x7f12c40a6540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56928/0 (socket says 172.21.15.121:56928) 2024-09-13T13:53:05.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.172+0000 7f12c8924640 1 -- 172.21.15.121:0/2752264508 learned_addr learned my addr 172.21.15.121:0/2752264508 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12c8924640 1 -- 172.21.15.121:0/2752264508 >> v1:172.21.15.121:6790/0 conn(0x7f12c409d150 legacy=0x7f12c413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12c9125640 1 -- 172.21.15.121:0/2752264508 >> v1:172.21.15.121:6793/0 conn(0x7f12c40a6160 legacy=0x7f12c40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368270688 0 0) 0x7f12c409d970 con 0x7f12c40a6160 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12bbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12bbfff640 10 monclient(hunting): my global_id is 4706 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.174+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12b0003520 con 0x7f12c40a6160 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1885716435 0 0) 0x7f12c409e210 con 0x7f12c409d150 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient(hunting): my global_id is 4682 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12c409d970 con 0x7f12c409d150 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12c9926640 1 -- 172.21.15.121:0/2752264508 >> v1:172.21.15.153:6791/0 conn(0x7f12c40a6c50 legacy=0x7f12c4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3960849484 0 0) 0x7f12b0003520 con 0x7f12c40a6160 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 >> v1:172.21.15.153:6791/0 conn(0x7f12c40a6c50 legacy=0x7f12c4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 >> v1:172.21.15.121:6790/0 conn(0x7f12c409d150 legacy=0x7f12c413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 monclient: found mon.noname-e 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c409e630 con 0x7f12c40a6160 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 5 monclient: authenticate success, global_id 4706 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12bc001e60 con 0x7f12c40a6160 2024-09-13T13:53:05.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: dump: 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.187 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12bc004b60 con 0x7f12c40a6160 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 1 -- 172.21.15.121:0/2752264508 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12bc0050d0 con 0x7f12c40a6160 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 4 set_mon_vals no callback set 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:05.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12bbfff640 10 monclient: dump: 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.189 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 10 monclient: shutdown 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 1 -- 172.21.15.121:0/2752264508 >> v1:172.21.15.121:6793/0 conn(0x7f12c40a6160 legacy=0x7f12c40a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 1 -- 172.21.15.121:0/2752264508 shutdown_connections 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 1 -- 172.21.15.121:0/2752264508 >> 172.21.15.121:0/2752264508 conn(0x7f12c40ab940 msgr2=0x7f12c40abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.175+0000 7f12cb389640 1 -- 172.21.15.121:0/2752264508 shutdown_connections 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.176+0000 7f12cb389640 1 -- 172.21.15.121:0/2752264508 wait complete. 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.176+0000 7f12cb389640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.176+0000 7f12cb389640 10 monclient: monmap: 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.190 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.176+0000 7f12cb389640 1 Processor -- start 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.176+0000 7f12cb389640 1 -- start start 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.176+0000 7f12cb389640 10 monclient: init 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient: _add_conns ranks=[5,3,8,1,0,2,4,7,6] 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient(hunting): picked mon.noname-h con 0x7f12c409d150 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient(hunting): picked mon.noname-g con 0x7f12c40a6160 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient(hunting): picked mon.noname-e con 0x7f12c40a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c40a46a0 con 0x7f12c40a6160 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c41247f0 con 0x7f12c409d150 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12c4126b50 con 0x7f12c40a6c50 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12cb389640 10 monclient(hunting): authenticate will time out at 864.807359s 2024-09-13T13:53:05.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12c9926640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f12c40a6c50 0x7f12c41d6240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56936/0 (socket says 172.21.15.121:56936) 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.177+0000 7f12c9926640 1 -- 172.21.15.121:0/1620963811 learned_addr learned my addr 172.21.15.121:0/1620963811 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12c9926640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.121:6793/0 conn(0x7f12c40a6c50 legacy=0x7f12c41d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12c9125640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.153:6791/0 conn(0x7f12c409d150 legacy=0x7f12c4139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12c8924640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.153:6790/0 conn(0x7f12c40a6160 legacy=0x7f12c41d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3184940186 0 0) 0x7f12c4126b50 con 0x7f12c40a6c50 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): my global_id is 4715 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12a8003520 con 0x7f12c40a6c50 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2671265556 0 0) 0x7f12c40a46a0 con 0x7f12c40a6160 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): my global_id is 4665 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12c4126b50 con 0x7f12c40a6160 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 906312835 0 0) 0x7f12c41247f0 con 0x7f12c409d150 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 10 monclient(hunting): my global_id is 4694 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.179+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12c40a46a0 con 0x7f12c409d150 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3508325910 0 0) 0x7f12a8003520 con 0x7f12c40a6c50 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12c41247f0 con 0x7f12c40a6c50 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c0003610 con 0x7f12c40a6c50 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.195 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 277978823 0 0) 0x7f12c41247f0 con 0x7f12c40a6c50 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.153:6791/0 conn(0x7f12c409d150 legacy=0x7f12c4139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.153:6790/0 conn(0x7f12c40a6160 legacy=0x7f12c41d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 monclient: found mon.e 2024-09-13T13:53:05.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c41262b0 con 0x7f12c40a6c50 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 5 monclient: authenticate success, global_id 4715 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 10 monclient: _renew_subs 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12c41398a0 con 0x7f12c40a6c50 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12c0003b10 con 0x7f12c40a6c50 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f12c0004080 con 0x7f12c40a6c50 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12b9ffb640 10 monclient: dump: 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.198 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12bb7fe640 4 set_mon_vals no callback set 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 10 monclient: _renew_subs 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.180+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7f12c41cd9d0 con 0x7f12c40a6c50 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f12c0004220 con 0x7f12c40a6c50 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12cb389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12cb389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12cb389640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c40a8b40 con 0x7f12c40a6c50 2024-09-13T13:53:05.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f12c004e1c0 con 0x7f12c40a6c50 2024-09-13T13:53:05.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.181+0000 7f12c9125640 10 monclient: get_auth_request con 0x7f12a803fd00 auth_method 0 2024-09-13T13:53:05.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.185+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f12c0017710 con 0x7f12c40a6c50 2024-09-13T13:53:05.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.185+0000 7f12b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.185+0000 7f12b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:05.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.359+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f12c41254d0 con 0x7f12a803fd00 2024-09-13T13:53:05.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.360+0000 7f12b9ffb640 1 -- 172.21.15.121:0/1620963811 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f12c41254d0 con 0x7f12a803fd00 2024-09-13T13:53:05.362 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.153:6812/1206688214 conn(0x7f12a803fd00 legacy=0x7f12a80421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 10 monclient: shutdown 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 >> v1:172.21.15.121:6793/0 conn(0x7f12c40a6c50 legacy=0x7f12c41d6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 shutdown_connections 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 >> 172.21.15.121:0/1620963811 conn(0x7f12c40ab940 msgr2=0x7f12c41c1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 shutdown_connections 2024-09-13T13:53:05.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.363+0000 7f12cb389640 1 -- 172.21.15.121:0/1620963811 wait complete. 2024-09-13T13:53:05.375 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T13:53:05.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T13:53:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.475+0000 7ff6e4409640 10 monclient: get_monmap_and_config 2024-09-13T13:53:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.475+0000 7ff6e4409640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.475+0000 7ff6e4409640 10 monclient: monmap: 2024-09-13T13:53:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:05.477364+0000 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:05.477364+0000 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.478 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 1 Processor -- start 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 1 -- start start 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient: init 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient: _add_conns ranks=[2,6,1,4,7,5,8,3,0] 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient(hunting): picked mon.noname-b con 0x7ff6dc0a5fc0 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient(hunting): picked mon.noname-d con 0x7ff6dc0a6ab0 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient(hunting): picked mon.noname-f con 0x7ff6dc09ddd0 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc0a71f0 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc0a7410 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc06f840 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.476+0000 7ff6e4409640 10 monclient(hunting): authenticate will time out at 865.106920s 2024-09-13T13:53:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.477+0000 7ff6e21a5640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7ff6dc0a5fc0 0x7ff6dc0a63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:49030/0 (socket says 172.21.15.121:49030) 2024-09-13T13:53:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.477+0000 7ff6e21a5640 1 -- 172.21.15.121:0/3702016508 learned_addr learned my addr 172.21.15.121:0/3702016508 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.477+0000 7ff6e19a4640 1 --1- 172.21.15.121:0/3702016508 >> v1:172.21.15.121:6792/0 conn(0x7ff6dc0a6ab0 0x7ff6dc13e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:44292/0 (socket says 172.21.15.121:44292) 2024-09-13T13:53:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e19a4640 1 -- 172.21.15.121:0/3702016508 >> v1:172.21.15.121:6792/0 conn(0x7ff6dc0a6ab0 legacy=0x7ff6dc13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e21a5640 1 -- 172.21.15.121:0/3702016508 >> v1:172.21.15.121:6790/0 conn(0x7ff6dc0a5fc0 legacy=0x7ff6dc0a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e29a6640 1 -- 172.21.15.121:0/3702016508 >> v1:172.21.15.153:6789/0 conn(0x7ff6dc09ddd0 legacy=0x7ff6dc141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775939539 0 0) 0x7ff6dc06f840 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): my global_id is 4785 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c8003520 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2676163386 0 0) 0x7ff6dc0a7410 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): my global_id is 4691 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6dc06f840 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3323588101 0 0) 0x7ff6dc0a71f0 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient(hunting): my global_id is 104239 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6dc0a7410 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 63516702 0 0) 0x7ff6dc06f840 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 >> v1:172.21.15.121:6792/0 conn(0x7ff6dc0a6ab0 legacy=0x7ff6dc13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 >> v1:172.21.15.153:6789/0 conn(0x7ff6dc09ddd0 legacy=0x7ff6dc141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 monclient: found mon.noname-b 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.121:6790/0 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc0a7720 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ff6d8001ac0 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e4409640 5 monclient: authenticate success, global_id 4691 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e4409640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e11a3640 10 monclient: dump: 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.487 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.479+0000 7ff6e4409640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.2 v1:172.21.15.121:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff6d80048a0 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e11a3640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e11a3640 1 -- 172.21.15.121:0/3702016508 <== mon.2 v1:172.21.15.121:6790/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ff6d8004e50 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:05.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 4 set_mon_vals no callback set 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e11a3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e11a3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e11a3640 10 monclient: dump: 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.489 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 10 monclient: shutdown 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 1 -- 172.21.15.121:0/3702016508 >> v1:172.21.15.121:6790/0 conn(0x7ff6dc0a5fc0 legacy=0x7ff6dc0a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 1 -- 172.21.15.121:0/3702016508 shutdown_connections 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 1 -- 172.21.15.121:0/3702016508 >> 172.21.15.121:0/3702016508 conn(0x7ff6dc06eab0 msgr2=0x7ff6dc06eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 1 -- 172.21.15.121:0/3702016508 shutdown_connections 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.480+0000 7ff6e4409640 1 -- 172.21.15.121:0/3702016508 wait complete. 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 10 monclient: monmap: 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.491 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 1 Processor -- start 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 1 -- start start 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 10 monclient: init 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 10 monclient: _add_conns ranks=[8,4,0,5,7,3,1,2,6] 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.481+0000 7ff6e4409640 10 monclient(hunting): picked mon.noname-e con 0x7ff6dc09ddd0 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 10 monclient(hunting): picked mon.noname-c con 0x7ff6dc0a5fc0 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 10 monclient(hunting): picked mon.noname-a con 0x7ff6dc0a6ab0 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc128cf0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc128fd0 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc0a5880 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e4409640 10 monclient(hunting): authenticate will time out at 865.112072s 2024-09-13T13:53:05.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e21a5640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7ff6dc09ddd0 0x7ff6dc141570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56948/0 (socket says 172.21.15.121:56948) 2024-09-13T13:53:05.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.482+0000 7ff6e21a5640 1 -- 172.21.15.121:0/2071445483 learned_addr learned my addr 172.21.15.121:0/2071445483 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6e19a4640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.121:6791/0 conn(0x7ff6dc0a5fc0 legacy=0x7ff6dc1d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6e29a6640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.121:6789/0 conn(0x7ff6dc0a6ab0 legacy=0x7ff6dc1d5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6e21a5640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.121:6793/0 conn(0x7ff6dc09ddd0 legacy=0x7ff6dc141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 552035583 0 0) 0x7ff6dc128fd0 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 4666 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6c0003550 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4253980599 0 0) 0x7ff6dc128cf0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 4743 2024-09-13T13:53:05.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6dc128fd0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331150568 0 0) 0x7ff6dc0a5880 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): my global_id is 4724 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6dc128cf0 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1873734222 0 0) 0x7ff6dc128fd0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6dc0a5880 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2431260868 0 0) 0x7ff6c0003550 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6dc128fd0 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ff6d4003990 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): dump: 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.499 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1858872807 0 0) 0x7ff6dc128cf0 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6c0003550 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ff6cc001d30 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): dump: 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.500 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.501 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ff6d8004d90 con 0x7ff6dc09ddd0 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:05.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient(hunting): dump: 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.503 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.504 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2503641258 0 0) 0x7ff6dc0a5880 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.121:6793/0 conn(0x7ff6dc09ddd0 legacy=0x7ff6dc141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.121:6791/0 conn(0x7ff6dc0a5fc0 legacy=0x7ff6dc1d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 1 monclient: found mon.a 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.484+0000 7ff6d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:05.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc128a10 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 5 monclient: authenticate success, global_id 4743 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1426850843 0 0) 0x7ff6dc128fd0 con 0x7ff6dc0a5fc0 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 10 monclient: _renew_subs 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:05.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc127ba0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 10 monclient: _renew_subs 2024-09-13T13:53:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1cd3a0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff6d4003ea0 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7ff6d4004390 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e09a2640 4 set_mon_vals no callback set 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6d2ffd640 10 monclient: dump: 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.508 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.121:6789/0 2024-09-13T13:53:05.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.485+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.121:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff6c4000f80 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7ff6d4004530 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7ff6d404e500 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.153:6804/3215471673 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6cc001d30 con 0x7ff6c0047110 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== mon.0 v1:172.21.15.121:6789/0 9 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (unknown 2135480514 0 0) 0x7ff6d404e970 con 0x7ff6dc0a6ab0 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6d2ffd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6e21a5640 10 monclient: get_auth_request con 0x7ff6c0040620 auth_method 0 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.486+0000 7ff6e19a4640 10 monclient: get_auth_request con 0x7ff6c0047110 auth_method 0 2024-09-13T13:53:05.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.492+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== osd.5 v1:172.21.15.153:6804/3215471673 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff6cc001d30 con 0x7ff6c0047110 2024-09-13T13:53:05.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.504+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 --> v1:172.21.15.153:6804/3215471673 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff6c4002cd0 con 0x7ff6c0047110 2024-09-13T13:53:05.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6d2ffd640 1 -- 172.21.15.121:0/2071445483 <== osd.5 v1:172.21.15.153:6804/3215471673 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7ff6c4002cd0 con 0x7ff6c0047110 2024-09-13T13:53:05.511 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-09-13T13:53:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T13:53:05.512 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.153:6804/3215471673 conn(0x7ff6c0047110 legacy=0x7ff6c0049530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.153:6812/1206688214 conn(0x7ff6c0040620 legacy=0x7ff6c0042ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 10 monclient: shutdown 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 >> v1:172.21.15.121:6789/0 conn(0x7ff6dc0a6ab0 legacy=0x7ff6dc1d5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e29a6640 1 -- 172.21.15.121:0/2071445483 reap_dead start 2024-09-13T13:53:05.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 shutdown_connections 2024-09-13T13:53:05.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 >> 172.21.15.121:0/2071445483 conn(0x7ff6dc06eab0 msgr2=0x7ff6dc0a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 shutdown_connections 2024-09-13T13:53:05.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.505+0000 7ff6e4409640 1 -- 172.21.15.121:0/2071445483 wait complete. 2024-09-13T13:53:05.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.616+0000 7f7f2739b640 10 monclient: get_monmap_and_config 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.616+0000 7f7f2739b640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.616+0000 7f7f2739b640 10 monclient: monmap: 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:05.617881+0000 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:05.617881+0000 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.618 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.619 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.616+0000 7f7f2739b640 1 Processor -- start 2024-09-13T13:53:05.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.616+0000 7f7f2739b640 1 -- start start 2024-09-13T13:53:05.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.616+0000 7f7f2739b640 10 monclient: init 2024-09-13T13:53:05.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient: _add_conns ranks=[5,0,8,2,6,3,1,4,7] 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient(hunting): picked mon.noname-h con 0x7f7f200a6160 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient(hunting): picked mon.noname-a con 0x7f7f2009d150 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient(hunting): picked mon.noname-e con 0x7f7f200a6c50 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f2009e210 con 0x7f7f2009d150 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f200a7ef0 con 0x7f7f200a6160 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f2009d970 con 0x7f7f200a6c50 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f2739b640 10 monclient(hunting): authenticate will time out at 865.247493s 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f25938640 1 --1- >> v1:172.21.15.121:6793/0 conn(0x7f7f200a6c50 0x7f7f20141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:56962/0 (socket says 172.21.15.121:56962) 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f24936640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7f2009d150 0x7f7f2013e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51620/0 (socket says 172.21.15.121:51620) 2024-09-13T13:53:05.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.617+0000 7f7f25938640 1 -- 172.21.15.121:0/532988771 learned_addr learned my addr 172.21.15.121:0/532988771 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f25938640 1 -- 172.21.15.121:0/532988771 >> v1:172.21.15.121:6793/0 conn(0x7f7f200a6c50 legacy=0x7f7f20141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f24936640 1 -- 172.21.15.121:0/532988771 >> v1:172.21.15.121:6789/0 conn(0x7f7f2009d150 legacy=0x7f7f2013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518092843 0 0) 0x7f7f2009d970 con 0x7f7f200a6c50 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 10 monclient(hunting): my global_id is 4733 2024-09-13T13:53:05.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f08003520 con 0x7f7f200a6c50 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.619+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940696202 0 0) 0x7f7f2009e210 con 0x7f7f2009d150 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient(hunting): my global_id is 4752 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f2009d970 con 0x7f7f2009d150 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f25137640 1 -- 172.21.15.121:0/532988771 >> v1:172.21.15.153:6791/0 conn(0x7f7f200a6160 legacy=0x7f7f200a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3249338769 0 0) 0x7f7f2009d970 con 0x7f7f2009d150 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 >> v1:172.21.15.121:6793/0 conn(0x7f7f200a6c50 legacy=0x7f7f20141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 >> v1:172.21.15.153:6791/0 conn(0x7f7f200a6160 legacy=0x7f7f200a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 monclient: found mon.noname-a 2024-09-13T13:53:05.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2009e630 con 0x7f7f2009d150 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 5 monclient: authenticate success, global_id 4752 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7f14001ac0 con 0x7f7f2009d150 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: dump: 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.627 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f140047e0 con 0x7f7f2009d150 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 1 -- 172.21.15.121:0/532988771 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7f14004d10 con 0x7f7f2009d150 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 4 set_mon_vals no callback set 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f1ffff640 10 monclient: dump: 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.629 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.630 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 10 monclient: shutdown 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 1 -- 172.21.15.121:0/532988771 >> v1:172.21.15.121:6789/0 conn(0x7f7f2009d150 legacy=0x7f7f2013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 1 -- 172.21.15.121:0/532988771 shutdown_connections 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.620+0000 7f7f2739b640 1 -- 172.21.15.121:0/532988771 >> 172.21.15.121:0/532988771 conn(0x7f7f200ab940 msgr2=0x7f7f200abd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 1 -- 172.21.15.121:0/532988771 shutdown_connections 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 1 -- 172.21.15.121:0/532988771 wait complete. 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 10 monclient: monmap: 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.631 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.632 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 1 Processor -- start 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 1 -- start start 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.621+0000 7f7f2739b640 10 monclient: init 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient: _add_conns ranks=[2,7,6,8,4,1,5,3,0] 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient(hunting): picked mon.noname-b con 0x7f7f2009d150 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient(hunting): picked mon.noname-i con 0x7f7f200a6160 addr v1:172.21.15.153:6792/0 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient(hunting): picked mon.noname-d con 0x7f7f200a6c50 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:05.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f20126870 con 0x7f7f2009d150 2024-09-13T13:53:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f20126590 con 0x7f7f200a6c50 2024-09-13T13:53:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 1 -- --> v1:172.21.15.153:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f201398a0 con 0x7f7f200a6160 2024-09-13T13:53:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f2739b640 10 monclient(hunting): authenticate will time out at 865.252368s 2024-09-13T13:53:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f25137640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f7f2009d150 0x7f7f20139190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:49040/0 (socket says 172.21.15.121:49040) 2024-09-13T13:53:05.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.622+0000 7f7f25137640 1 -- 172.21.15.121:0/368085593 learned_addr learned my addr 172.21.15.121:0/368085593 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f25137640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.121:6790/0 conn(0x7f7f2009d150 legacy=0x7f7f20139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f25938640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.121:6792/0 conn(0x7f7f200a6c50 legacy=0x7f7f201d60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3880700501 0 0) 0x7f7f20126870 con 0x7f7f2009d150 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): my global_id is 4700 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f0c003550 con 0x7f7f2009d150 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1124916 0 0) 0x7f7f20126590 con 0x7f7f200a6c50 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): my global_id is 4794 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f20126870 con 0x7f7f200a6c50 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f24936640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.153:6792/0 conn(0x7f7f200a6160 legacy=0x7f7f201d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3115941570 0 0) 0x7f7f0c003550 con 0x7f7f2009d150 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f20126590 con 0x7f7f2009d150 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7f10003990 con 0x7f7f2009d150 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): dump: 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.639 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.640 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.624+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3665025531 0 0) 0x7f7f20126870 con 0x7f7f200a6c50 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f0c003550 con 0x7f7f200a6c50 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7f18001d30 con 0x7f7f200a6c50 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): dump: 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.641 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.7 v1:172.21.15.153:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 70148822 0 0) 0x7f7f201398a0 con 0x7f7f200a6160 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient(hunting): my global_id is 104272 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.153:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f20126870 con 0x7f7f200a6160 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 36728400 0 0) 0x7f7f20126590 con 0x7f7f2009d150 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.153:6792/0 conn(0x7f7f200a6160 legacy=0x7f7f201d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.121:6792/0 conn(0x7f7f200a6c50 legacy=0x7f7f201d60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 monclient: found mon.b 2024-09-13T13:53:05.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f20139a70 con 0x7f7f2009d150 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 5 monclient: authenticate success, global_id 4700 2024-09-13T13:53:05.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 10 monclient: _renew_subs 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f201cd680 con 0x7f7f2009d150 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f10003f10 con 0x7f7f2009d150 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f7f10004440 con 0x7f7f2009d150 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1dffb640 10 monclient: dump: 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.645 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.646 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 10 monclient: _renew_subs 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f201d77f0 con 0x7f7f2009d150 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.625+0000 7f7f1f7fe640 4 set_mon_vals no callback set 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f7f100045e0 con 0x7f7f2009d150 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f2739b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f2739b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f2739b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f201247f0 con 0x7f7f2009d150 2024-09-13T13:53:05.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f7f1004e4e0 con 0x7f7f2009d150 2024-09-13T13:53:05.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.626+0000 7f7f24936640 10 monclient: get_auth_request con 0x7f7f0c040470 auth_method 0 2024-09-13T13:53:05.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.630+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f7f201247f0 con 0x7f7f2009d150 2024-09-13T13:53:05.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.630+0000 7f7f1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.630+0000 7f7f1dffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:05.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.806+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f7f200a4540 con 0x7f7f0c040470 2024-09-13T13:53:05.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.807+0000 7f7f1dffb640 1 -- 172.21.15.121:0/368085593 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f7f200a4540 con 0x7f7f0c040470 2024-09-13T13:53:05.809 INFO:teuthology.orchestra.run.smithi121.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T13:53:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.153:6812/1206688214 conn(0x7f7f0c040470 legacy=0x7f7f0c042930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 10 monclient: shutdown 2024-09-13T13:53:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 >> v1:172.21.15.121:6790/0 conn(0x7f7f2009d150 legacy=0x7f7f20139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 shutdown_connections 2024-09-13T13:53:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 >> 172.21.15.121:0/368085593 conn(0x7f7f200ab940 msgr2=0x7f7f200aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 shutdown_connections 2024-09-13T13:53:05.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.809+0000 7f7f2739b640 1 -- 172.21.15.121:0/368085593 wait complete. 2024-09-13T13:53:05.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:53:05.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.917+0000 7fa54c1ec640 10 monclient: get_monmap_and_config 2024-09-13T13:53:05.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.917+0000 7fa54c1ec640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.917+0000 7fa54c1ec640 10 monclient: monmap: 2024-09-13T13:53:05.919 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:05.919374+0000 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:05.919374+0000 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.920 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.921 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.921 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.921 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.921 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.921 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.921 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 1 Processor -- start 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 1 -- start start 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient: init 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient: _add_conns ranks=[0,3,2,6,7,4,5,1,8] 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient(hunting): picked mon.noname-a con 0x7fa54409c950 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient(hunting): picked mon.noname-g con 0x7fa544135b50 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient(hunting): picked mon.noname-b con 0x7fa54413f660 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa544135f50 con 0x7fa54409c950 2024-09-13T13:53:05.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54409d860 con 0x7fa54413f660 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54409db40 con 0x7fa544135b50 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.918+0000 7fa54c1ec640 10 monclient(hunting): authenticate will time out at 865.548950s 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.919+0000 7fa54a789640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fa54413f660 0x7fa544141a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:49052/0 (socket says 172.21.15.121:49052) 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.919+0000 7fa549f88640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa54409c950 0x7fa5440a7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:51632/0 (socket says 172.21.15.121:51632) 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.919+0000 7fa54a789640 1 -- 172.21.15.121:0/639788773 learned_addr learned my addr 172.21.15.121:0/639788773 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa549f88640 1 -- 172.21.15.121:0/639788773 >> v1:172.21.15.121:6789/0 conn(0x7fa54409c950 legacy=0x7fa5440a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa54a789640 1 -- 172.21.15.121:0/639788773 >> v1:172.21.15.121:6790/0 conn(0x7fa54413f660 legacy=0x7fa544141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260953010 0 0) 0x7fa54409d860 con 0x7fa54413f660 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): my global_id is 4709 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa530003520 con 0x7fa54413f660 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900002233 0 0) 0x7fa544135f50 con 0x7fa54409c950 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient(hunting): my global_id is 4761 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa54409d860 con 0x7fa54409c950 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa549787640 1 -- 172.21.15.121:0/639788773 >> v1:172.21.15.153:6790/0 conn(0x7fa544135b50 legacy=0x7fa54413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 <== mon.0 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 489860258 0 0) 0x7fa54409d860 con 0x7fa54409c950 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 >> v1:172.21.15.153:6790/0 conn(0x7fa544135b50 legacy=0x7fa54413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 >> v1:172.21.15.121:6790/0 conn(0x7fa54413f660 legacy=0x7fa544141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 monclient: found mon.noname-a 2024-09-13T13:53:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.121:6789/0 2024-09-13T13:53:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54409de20 con 0x7fa54409c950 2024-09-13T13:53:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa54c1ec640 5 monclient: authenticate success, global_id 4761 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 <== mon.0 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa540001a40 con 0x7fa54409c950 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa54c1ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: dump: 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.931 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa54c1ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:05.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 <== mon.0 v1:172.21.15.121:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa540001d10 con 0x7fa54409c950 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 1 -- 172.21.15.121:0/639788773 <== mon.0 v1:172.21.15.121:6789/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa540004b20 con 0x7fa54409c950 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa54c1ec640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa54c1ec640 4 set_mon_vals no callback set 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.921+0000 7fa548f86640 10 monclient: dump: 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.933 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 10 monclient: shutdown 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 1 -- 172.21.15.121:0/639788773 >> v1:172.21.15.121:6789/0 conn(0x7fa54409c950 legacy=0x7fa5440a7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 1 -- 172.21.15.121:0/639788773 shutdown_connections 2024-09-13T13:53:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 1 -- 172.21.15.121:0/639788773 >> 172.21.15.121:0/639788773 conn(0x7fa5440ab760 msgr2=0x7fa5440abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 1 -- 172.21.15.121:0/639788773 shutdown_connections 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 1 -- 172.21.15.121:0/639788773 wait complete. 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 10 monclient: build_initial_monmap 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.922+0000 7fa54c1ec640 10 monclient: monmap: 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 1 Processor -- start 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 1 -- start start 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient: init 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient: _add_conns ranks=[2,8,1,5,0,6,7,4,3] 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient(hunting): picked mon.noname-b con 0x7fa54409c950 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient(hunting): picked mon.noname-e con 0x7fa544135b50 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient(hunting): picked mon.noname-f con 0x7fa54413f660 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54413bcb0 con 0x7fa54413f660 2024-09-13T13:53:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54413dd10 con 0x7fa54409c950 2024-09-13T13:53:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa544135390 con 0x7fa544135b50 2024-09-13T13:53:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa54c1ec640 10 monclient(hunting): authenticate will time out at 865.553750s 2024-09-13T13:53:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa549f88640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7fa54409c950 0x7fa54413a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:49056/0 (socket says 172.21.15.121:49056) 2024-09-13T13:53:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.923+0000 7fa549f88640 1 -- 172.21.15.121:0/2440745718 learned_addr learned my addr 172.21.15.121:0/2440745718 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.925+0000 7fa549f88640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.121:6790/0 conn(0x7fa54409c950 legacy=0x7fa54413a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.925+0000 7fa549787640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.121:6793/0 conn(0x7fa544135b50 legacy=0x7fa54413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa549f88640 1 --1- 172.21.15.121:0/2440745718 >> v1:172.21.15.121:6790/0 conn(0x7fa54409c950 0x7fa54413a7b0 :-1 s=OPENED pgs=184 cs=1 l=1).queue_received will delay after 1.72624e+09 on 0x7fa54409de20 auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1620852726 0 0) 0x7fa544135390 con 0x7fa544135b50 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): my global_id is 4742 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa528003520 con 0x7fa544135b50 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54a789640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.153:6789/0 conn(0x7fa54413f660 legacy=0x7fa5441d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 147540193 0 0) 0x7fa528003520 con 0x7fa544135b50 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.121:6793/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa544135390 con 0x7fa544135b50 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa534003610 con 0x7fa544135b50 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): dump: 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.944 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.945 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2116773218 0 0) 0x7fa54413bcb0 con 0x7fa54413f660 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _negotiate 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient(hunting): my global_id is 104248 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa528003520 con 0x7fa54413f660 2024-09-13T13:53:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4230095740 0 0) 0x7fa544135390 con 0x7fa544135b50 2024-09-13T13:53:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.121:6790/0 conn(0x7fa54409c950 legacy=0x7fa54413a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.153:6789/0 conn(0x7fa54413f660 legacy=0x7fa5441d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: _finish_hunting 0 2024-09-13T13:53:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 monclient: found mon.e 2024-09-13T13:53:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54413c250 con 0x7fa544135b50 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: _finish_auth 0 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: _check_auth_tickets 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 5 monclient: authenticate success, global_id 4742 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261757640 0 0) 0x7fa54409de20 con 0x7fa54409c950 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 10 monclient: _renew_subs 2024-09-13T13:53:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.121:6793/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa54413b5d0 con 0x7fa544135b50 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 10 monclient: _renew_subs 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.121:6793/0 -- mon_subscribe({osdmap=0}) -- 0x7fa54413d050 con 0x7fa544135b50 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa534003ba0 con 0x7fa544135b50 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7fa534004090 con 0x7fa544135b50 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53bfff640 4 set_mon_vals no callback set 2024-09-13T13:53:05.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.926+0000 7fa53a7fc640 10 monclient: dump: 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:05.950 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7fa534004330 con 0x7fa544135b50 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7fa53404e310 con 0x7fa544135b50 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa54c1ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa54c1ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa54c1ec640 10 monclient: _send_mon_message to mon.e at v1:172.21.15.121:6793/0 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.121:6793/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa52c005080 con 0x7fa544135b50 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.927+0000 7fa549f88640 10 monclient: get_auth_request con 0x7fa52803fb10 auth_method 0 2024-09-13T13:53:05.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.932+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mon.8 v1:172.21.15.121:6793/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7fa534017920 con 0x7fa544135b50 2024-09-13T13:53:05.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.932+0000 7fa53a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:05.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:05.932+0000 7fa53a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:06.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.101+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa52c000b90 con 0x7fa52803fb10 2024-09-13T13:53:06.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.103+0000 7fa53a7fc640 1 -- 172.21.15.121:0/2440745718 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33967 (unknown 2965378022 0 1221036938) 0x7fa540007e50 con 0x7fa52803fb10 2024-09-13T13:53:06.105 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:53:06.105 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.153:6812/1206688214 conn(0x7fa52803fb10 legacy=0x7fa528041fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 10 monclient: shutdown 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 >> v1:172.21.15.121:6793/0 conn(0x7fa544135b50 legacy=0x7fa54413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 shutdown_connections 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 >> 172.21.15.121:0/2440745718 conn(0x7fa5440ab760 msgr2=0x7fa544134850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 shutdown_connections 2024-09-13T13:53:06.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:06.105+0000 7fa54c1ec640 1 -- 172.21.15.121:0/2440745718 wait complete. 2024-09-13T13:53:06.117 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-13T13:53:05.236091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163668,"kb_used_data":1504,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067372,"statfs":{"total":579820584960,"available":579652988928,"internally_reserved":0,"allocated":1540096,"data_stored":1560928,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002479"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.401992+0000","last_change":"2024-09-13T13:51:43.470385+0000","last_active":"2024-09-13T13:52:08.401992+0000","last_peered":"2024-09-13T13:52:08.401992+0000","last_clean":"2024-09-13T13:52:08.401992+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:52:08.401992+0000","last_undegraded":"2024-09-13T13:52:08.401992+0000","last_fullsized":"2024-09-13T13:52:08.401992+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:43.260614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.169843+0000","last_change":"2024-09-13T13:51:43.470282+0000","last_active":"2024-09-13T13:52:09.169843+0000","last_peered":"2024-09-13T13:52:09.169843+0000","last_clean":"2024-09-13T13:52:09.169843+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:52:09.169843+0000","last_undegraded":"2024-09-13T13:52:09.169843+0000","last_fullsized":"2024-09-13T13:52:09.169843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:12.108672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402216+0000","last_change":"2024-09-13T13:51:43.470404+0000","last_active":"2024-09-13T13:52:08.402216+0000","last_peered":"2024-09-13T13:52:08.402216+0000","last_clean":"2024-09-13T13:52:08.402216+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:52:08.402216+0000","last_undegraded":"2024-09-13T13:52:08.402216+0000","last_fullsized":"2024-09-13T13:52:08.402216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:11:25.708801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:09.171472+0000","last_change":"2024-09-13T13:51:43.470339+0000","last_active":"2024-09-13T13:52:09.171472+0000","last_peered":"2024-09-13T13:52:09.171472+0000","last_clean":"2024-09-13T13:52:09.171472+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:52:09.171472+0000","last_undegraded":"2024-09-13T13:52:09.171472+0000","last_fullsized":"2024-09-13T13:52:09.171472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:54.334976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":36,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403901+0000","last_change":"2024-09-13T13:51:43.471963+0000","last_active":"2024-09-13T13:52:08.403901+0000","last_peered":"2024-09-13T13:52:08.403901+0000","last_clean":"2024-09-13T13:52:08.403901+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:52:08.403901+0000","last_undegraded":"2024-09-13T13:52:08.403901+0000","last_fullsized":"2024-09-13T13:52:08.403901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:03.301040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.404547+0000","last_change":"2024-09-13T13:51:43.467989+0000","last_active":"2024-09-13T13:52:08.404547+0000","last_peered":"2024-09-13T13:52:08.404547+0000","last_clean":"2024-09-13T13:52:08.404547+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:52:08.404547+0000","last_undegraded":"2024-09-13T13:52:08.404547+0000","last_fullsized":"2024-09-13T13:52:08.404547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:15.161755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402141+0000","last_change":"2024-09-13T13:51:43.470984+0000","last_active":"2024-09-13T13:52:08.402141+0000","last_peered":"2024-09-13T13:52:08.402141+0000","last_clean":"2024-09-13T13:52:08.402141+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:52:08.402141+0000","last_undegraded":"2024-09-13T13:52:08.402141+0000","last_fullsized":"2024-09-13T13:52:08.402141+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:02.788502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":35,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.403421+0000","last_change":"2024-09-13T13:51:43.471333+0000","last_active":"2024-09-13T13:52:08.403421+0000","last_peered":"2024-09-13T13:52:08.403421+0000","last_clean":"2024-09-13T13:52:08.403421+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:52:08.403421+0000","last_undegraded":"2024-09-13T13:52:08.403421+0000","last_fullsized":"2024-09-13T13:52:08.403421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:40.441552+0000","last_clean_scrub_stamp":"2024-09-13T13:51:40.441552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T14:39:10.213946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":125,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:52:08.402247+0000","last_change":"2024-09-13T13:51:37.384364+0000","last_active":"2024-09-13T13:52:08.402247+0000","last_peered":"2024-09-13T13:52:08.402247+0000","last_clean":"2024-09-13T13:52:08.402247+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:52:08.402247+0000","last_undegraded":"2024-09-13T13:52:08.402247+0000","last_fullsized":"2024-09-13T13:52:08.402247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:51:36.267919+0000","last_clean_scrub_stamp":"2024-09-13T13:51:36.267919+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:55.812981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607572,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607573,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:53:06.121 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-13T13:51:40.441552+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=40, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T13:51:36.267919+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=51, tm_sec=36, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=13, tm_min=53, tm_sec=2, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T13:53:06.122 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T13:53:26.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T13:53:26.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.232+0000 7f6ec11e2640 10 monclient: get_monmap_and_config 2024-09-13T13:53:26.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.232+0000 7f6ec11e2640 10 monclient: build_initial_monmap 2024-09-13T13:53:26.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.232+0000 7f6ec11e2640 10 monclient: monmap: 2024-09-13T13:53:26.233 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:26.233 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:26.233363+0000 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:26.233363+0000 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:26.234 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 1 Processor -- start 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 1 -- start start 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient: init 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient: _add_conns ranks=[3,5,4,1,2,7,0,8,6] 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient(hunting): picked mon.noname-g con 0x7f6ebc1390f0 addr v1:172.21.15.153:6790/0 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient(hunting): picked mon.noname-h con 0x7f6ebc09f1a0 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6ebc09e590 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 1 -- --> v1:172.21.15.153:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc09fab0 con 0x7f6ebc1390f0 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc139be0 con 0x7f6ebc09e590 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc09d960 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:26.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.233+0000 7f6ec11e2640 10 monclient(hunting): authenticate will time out at 885.862953s 2024-09-13T13:53:26.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.234+0000 7f6ebb59e640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f6ebc09e590 0x7f6ebc1419a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33462/0 (socket says 172.21.15.121:33462) 2024-09-13T13:53:26.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.234+0000 7f6ebb59e640 1 -- 172.21.15.121:0/1459411966 learned_addr learned my addr 172.21.15.121:0/1459411966 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6ebb59e640 1 -- 172.21.15.121:0/1459411966 >> v1:172.21.15.121:6791/0 conn(0x7f6ebc09e590 legacy=0x7f6ebc1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6ebad9d640 1 -- 172.21.15.121:0/1459411966 >> v1:172.21.15.153:6790/0 conn(0x7f6ebc1390f0 legacy=0x7f6ebc1394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eb3fff640 1 -- 172.21.15.121:0/1459411966 >> v1:172.21.15.153:6791/0 conn(0x7f6ebc09f1a0 legacy=0x7f6ebc13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3251438078 0 0) 0x7f6ebc139be0 con 0x7f6ebc09e590 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): my global_id is 4675 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea8003520 con 0x7f6ebc09e590 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.3 v1:172.21.15.153:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3352048807 0 0) 0x7f6ebc09fab0 con 0x7f6ebc1390f0 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): my global_id is 4683 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 --> v1:172.21.15.153:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ebc139be0 con 0x7f6ebc1390f0 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 396145840 0 0) 0x7f6ebc09d960 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient(hunting): my global_id is 4712 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ebc09fab0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1254979174 0 0) 0x7f6ea8003520 con 0x7f6ebc09e590 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 >> v1:172.21.15.153:6791/0 conn(0x7f6ebc09f1a0 legacy=0x7f6ebc13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 >> v1:172.21.15.153:6790/0 conn(0x7f6ebc1390f0 legacy=0x7f6ebc1394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: _finish_hunting 0 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 monclient: found mon.noname-c 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.121:6791/0 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc09db30 con 0x7f6ebc09e590 2024-09-13T13:53:26.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: _finish_auth 0 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: _check_auth_tickets 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6ec11e2640 5 monclient: authenticate success, global_id 4675 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6eb4001e60 con 0x7f6ebc09e590 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.236+0000 7f6eba59c640 10 monclient: dump: 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.4 v1:172.21.15.121:6791/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6eb4004c20 con 0x7f6ebc09e590 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6eba59c640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6eba59c640 1 -- 172.21.15.121:0/1459411966 <== mon.4 v1:172.21.15.121:6791/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6eb4005190 con 0x7f6ebc09e590 2024-09-13T13:53:26.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 4 set_mon_vals no callback set 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6eba59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6eba59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6eba59c640 10 monclient: dump: 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.246 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 10 monclient: shutdown 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 1 -- 172.21.15.121:0/1459411966 >> v1:172.21.15.121:6791/0 conn(0x7f6ebc09e590 legacy=0x7f6ebc1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 1 -- 172.21.15.121:0/1459411966 shutdown_connections 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 1 -- 172.21.15.121:0/1459411966 >> 172.21.15.121:0/1459411966 conn(0x7f6ebc069450 msgr2=0x7f6ebc0ab660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 1 -- 172.21.15.121:0/1459411966 shutdown_connections 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.237+0000 7f6ec11e2640 1 -- 172.21.15.121:0/1459411966 wait complete. 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.238+0000 7f6ec11e2640 10 monclient: build_initial_monmap 2024-09-13T13:53:26.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.238+0000 7f6ec11e2640 10 monclient: monmap: 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:26.248 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.238+0000 7f6ec11e2640 1 Processor -- start 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.238+0000 7f6ec11e2640 1 -- start start 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.238+0000 7f6ec11e2640 10 monclient: init 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient: _add_conns ranks=[6,2,1,3,5,0,7,8,4] 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient(hunting): picked mon.noname-d con 0x7f6ebc09e590 addr v1:172.21.15.121:6792/0 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6ebc09f1a0 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient(hunting): picked mon.noname-f con 0x7f6ebc1390f0 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc1420d0 con 0x7f6ebc1390f0 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc140490 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 1 -- --> v1:172.21.15.121:6792/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ebc13e130 con 0x7f6ebc09e590 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:26.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ec11e2640 10 monclient(hunting): authenticate will time out at 885.868314s 2024-09-13T13:53:26.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6eb3fff640 1 --1- >> v1:172.21.15.121:6790/0 conn(0x7f6ebc09f1a0 0x7f6ebc1d2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6790/0 says I am v1:172.21.15.121:52372/0 (socket says 172.21.15.121:52372) 2024-09-13T13:53:26.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6ebad9d640 1 --1- >> v1:172.21.15.121:6792/0 conn(0x7f6ebc09e590 0x7f6ebc1419c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6792/0 says I am v1:172.21.15.121:48392/0 (socket says 172.21.15.121:48392) 2024-09-13T13:53:26.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.239+0000 7f6eb3fff640 1 -- 172.21.15.121:0/967735685 learned_addr learned my addr 172.21.15.121:0/967735685 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb3fff640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.121:6790/0 conn(0x7f6ebc09f1a0 legacy=0x7f6ebc1d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6ebad9d640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.121:6792/0 conn(0x7f6ebc09e590 legacy=0x7f6ebc1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3940285858 0 0) 0x7f6ebc140490 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): my global_id is 4727 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e9c003550 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.6 v1:172.21.15.121:6792/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 664398336 0 0) 0x7f6ebc13e130 con 0x7f6ebc09e590 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): my global_id is 4803 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6792/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ebc140490 con 0x7f6ebc09e590 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6ebb59e640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.153:6789/0 conn(0x7f6ebc1390f0 legacy=0x7f6ebc1d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2367312473 0 0) 0x7f6ebc1420d0 con 0x7f6ebc1390f0 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): my global_id is 104257 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ebc13e130 con 0x7f6ebc1390f0 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2423999115 0 0) 0x7f6e9c003550 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6ebc1420d0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6ea4003610 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): dump: 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.255 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.6 v1:172.21.15.121:6792/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2868491805 0 0) 0x7f6ebc140490 con 0x7f6ebc09e590 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6792/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6e9c003550 con 0x7f6ebc09e590 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.241+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.6 v1:172.21.15.121:6792/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6eac001cf0 con 0x7f6ebc09e590 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient(hunting): got monmap 1 from mon.d (according to old e1) 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient(hunting): dump: 2024-09-13T13:53:26.256 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.257 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3298038054 0 0) 0x7f6ebc1420d0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.121:6792/0 conn(0x7f6ebc09e590 legacy=0x7f6ebc1419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.153:6789/0 conn(0x7f6ebc1390f0 legacy=0x7f6ebc1d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: _finish_hunting 0 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 monclient: found mon.b 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc13f0d0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: _finish_auth 0 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: _check_auth_tickets 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 5 monclient: authenticate success, global_id 4727 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 10 monclient: _renew_subs 2024-09-13T13:53:26.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6ebc13fbf0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 10 monclient: _renew_subs 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6ebc13ee10 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6ea4003bd0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6ea4004140 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb9d9b640 4 set_mon_vals no callback set 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.242+0000 7f6eb37fe640 10 monclient: dump: 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.259 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.260 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f6ea40042e0 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6ec11e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6ec11e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6ec11e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.121:6790/0 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.121:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ebc13f860 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f6ea404e260 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.243+0000 7f6ebad9d640 10 monclient: get_auth_request con 0x7f6e9c040030 auth_method 0 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.248+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mon.2 v1:172.21.15.121:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f6ea4017830 con 0x7f6ebc09f1a0 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.248+0000 7f6eb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:26.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.248+0000 7f6eb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:26.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.426+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 --> v1:172.21.15.153:6812/1206688214 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6ebc0a0350 con 0x7f6e9c040030 2024-09-13T13:53:26.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.427+0000 7f6eb37fe640 1 -- 172.21.15.121:0/967735685 <== mgr.4121 v1:172.21.15.153:6812/1206688214 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33975 (unknown 2965378022 0 991432027) 0x7f6ebc0a0350 con 0x7f6e9c040030 2024-09-13T13:53:26.429 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-09-13T13:53:26.429 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-09-13T13:53:26.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.429+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.153:6812/1206688214 conn(0x7f6e9c040030 legacy=0x7f6e9c0424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.429+0000 7f6ec11e2640 10 monclient: shutdown 2024-09-13T13:53:26.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.429+0000 7f6ec11e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:26.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.429+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 >> v1:172.21.15.121:6790/0 conn(0x7f6ebc09f1a0 legacy=0x7f6ebc1d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.429+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 shutdown_connections 2024-09-13T13:53:26.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.429+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 >> 172.21.15.121:0/967735685 conn(0x7f6ebc069450 msgr2=0x7f6ebc09d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:26.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.430+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 shutdown_connections 2024-09-13T13:53:26.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.430+0000 7f6ec11e2640 1 -- 172.21.15.121:0/967735685 wait complete. 2024-09-13T13:53:26.441 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-13T13:53:25.240504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163652,"kb_used_data":1504,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067388,"statfs":{"total":579820584960,"available":579653005312,"internally_reserved":0,"allocated":1540096,"data_stored":1560928,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002579"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:05.391203+0000","last_change":"2024-09-13T13:53:05.391203+0000","last_active":"2024-09-13T13:53:05.391203+0000","last_peered":"2024-09-13T13:53:05.391203+0000","last_clean":"2024-09-13T13:53:05.391203+0000","last_became_active":"2024-09-13T13:51:41.925410+0000","last_became_peered":"2024-09-13T13:51:41.925410+0000","last_unstale":"2024-09-13T13:53:05.391203+0000","last_undegraded":"2024-09-13T13:53:05.391203+0000","last_fullsized":"2024-09-13T13:53:05.391203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:53:05.391119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:53:05.391119+0000","last_clean_scrub_stamp":"2024-09-13T13:53:05.391119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:31.587130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065199400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:04.051021+0000","last_change":"2024-09-13T13:53:04.051021+0000","last_active":"2024-09-13T13:53:04.051021+0000","last_peered":"2024-09-13T13:53:04.051021+0000","last_clean":"2024-09-13T13:53:04.051021+0000","last_became_active":"2024-09-13T13:51:41.452866+0000","last_became_peered":"2024-09-13T13:51:41.452866+0000","last_unstale":"2024-09-13T13:53:04.051021+0000","last_undegraded":"2024-09-13T13:53:04.051021+0000","last_fullsized":"2024-09-13T13:53:04.051021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:53:04.050955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:53:04.050955+0000","last_clean_scrub_stamp":"2024-09-13T13:53:04.050955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:10.667762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041909400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:06.373848+0000","last_change":"2024-09-13T13:53:06.373848+0000","last_active":"2024-09-13T13:53:06.373848+0000","last_peered":"2024-09-13T13:53:06.373848+0000","last_clean":"2024-09-13T13:53:06.373848+0000","last_became_active":"2024-09-13T13:51:41.453041+0000","last_became_peered":"2024-09-13T13:51:41.453041+0000","last_unstale":"2024-09-13T13:53:06.373848+0000","last_undegraded":"2024-09-13T13:53:06.373848+0000","last_fullsized":"2024-09-13T13:53:06.373848+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:53:06.373773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:53:06.373773+0000","last_clean_scrub_stamp":"2024-09-13T13:53:06.373773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:13.209244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000553923,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:05.049028+0000","last_change":"2024-09-13T13:53:05.049028+0000","last_active":"2024-09-13T13:53:05.049028+0000","last_peered":"2024-09-13T13:53:05.049028+0000","last_clean":"2024-09-13T13:53:05.049028+0000","last_became_active":"2024-09-13T13:51:41.454560+0000","last_became_peered":"2024-09-13T13:51:41.454560+0000","last_unstale":"2024-09-13T13:53:05.049028+0000","last_undegraded":"2024-09-13T13:53:05.049028+0000","last_fullsized":"2024-09-13T13:53:05.049028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:53:05.048959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:53:05.048959+0000","last_clean_scrub_stamp":"2024-09-13T13:53:05.048959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:41.781309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050103400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"27'3","reported_seq":45,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:06.328485+0000","last_change":"2024-09-13T13:53:06.328485+0000","last_active":"2024-09-13T13:53:06.328485+0000","last_peered":"2024-09-13T13:53:06.328485+0000","last_clean":"2024-09-13T13:53:06.328485+0000","last_became_active":"2024-09-13T13:51:41.460300+0000","last_became_peered":"2024-09-13T13:51:41.460300+0000","last_unstale":"2024-09-13T13:53:06.328485+0000","last_undegraded":"2024-09-13T13:53:06.328485+0000","last_fullsized":"2024-09-13T13:53:06.328485+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'3","last_scrub_stamp":"2024-09-13T13:53:06.328376+0000","last_deep_scrub":"27'3","last_deep_scrub_stamp":"2024-09-13T13:53:06.328376+0000","last_clean_scrub_stamp":"2024-09-13T13:53:06.328376+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:50.136284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012115419999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:04.441074+0000","last_change":"2024-09-13T13:53:04.441074+0000","last_active":"2024-09-13T13:53:04.441074+0000","last_peered":"2024-09-13T13:53:04.441074+0000","last_clean":"2024-09-13T13:53:04.441074+0000","last_became_active":"2024-09-13T13:51:41.455697+0000","last_became_peered":"2024-09-13T13:51:41.455697+0000","last_unstale":"2024-09-13T13:53:04.441074+0000","last_undegraded":"2024-09-13T13:53:04.441074+0000","last_fullsized":"2024-09-13T13:53:04.441074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:53:04.440998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:53:04.440998+0000","last_clean_scrub_stamp":"2024-09-13T13:53:04.440998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:56.878514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056601100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:07.386517+0000","last_change":"2024-09-13T13:53:07.386517+0000","last_active":"2024-09-13T13:53:07.386517+0000","last_peered":"2024-09-13T13:53:07.386517+0000","last_clean":"2024-09-13T13:53:07.386517+0000","last_became_active":"2024-09-13T13:51:41.452365+0000","last_became_peered":"2024-09-13T13:51:41.452365+0000","last_unstale":"2024-09-13T13:53:07.386517+0000","last_undegraded":"2024-09-13T13:53:07.386517+0000","last_fullsized":"2024-09-13T13:53:07.386517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T13:53:07.386408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T13:53:07.386408+0000","last_clean_scrub_stamp":"2024-09-13T13:53:07.386408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:30.339481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048463100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"27'2","reported_seq":44,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:07.370495+0000","last_change":"2024-09-13T13:53:07.370495+0000","last_active":"2024-09-13T13:53:07.370495+0000","last_peered":"2024-09-13T13:53:07.370495+0000","last_clean":"2024-09-13T13:53:07.370495+0000","last_became_active":"2024-09-13T13:51:41.453842+0000","last_became_peered":"2024-09-13T13:51:41.453842+0000","last_unstale":"2024-09-13T13:53:07.370495+0000","last_undegraded":"2024-09-13T13:53:07.370495+0000","last_fullsized":"2024-09-13T13:53:07.370495+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-09-13T13:53:07.370401+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-09-13T13:53:07.370401+0000","last_clean_scrub_stamp":"2024-09-13T13:53:07.370401+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:02.591167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056330599999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'74","reported_seq":135,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-13T13:53:08.390738+0000","last_change":"2024-09-13T13:53:08.390738+0000","last_active":"2024-09-13T13:53:08.390738+0000","last_peered":"2024-09-13T13:53:08.390738+0000","last_clean":"2024-09-13T13:53:08.390738+0000","last_became_active":"2024-09-13T13:51:37.384019+0000","last_became_peered":"2024-09-13T13:51:37.384019+0000","last_unstale":"2024-09-13T13:53:08.390738+0000","last_undegraded":"2024-09-13T13:53:08.390738+0000","last_fullsized":"2024-09-13T13:53:08.390738+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'74","last_scrub_stamp":"2024-09-13T13:53:08.390645+0000","last_deep_scrub":"27'74","last_deep_scrub_stamp":"2024-09-13T13:53:08.390645+0000","last_clean_scrub_stamp":"2024-09-13T13:53:08.390645+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T15:39:15.970353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":11034,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":14,"seq":60129542168,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607576,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":83839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":290816,"data_stored":612767,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5517,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T13:53:26.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.544+0000 7f619f4d4640 10 monclient: get_monmap_and_config 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.544+0000 7f619f4d4640 10 monclient: build_initial_monmap 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.544+0000 7f619f4d4640 10 monclient: monmap: 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:53:26.545070+0000 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:53:26.545070+0000 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:26.545 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:26.546 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.544+0000 7f619f4d4640 1 Processor -- start 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 1 -- start start 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient: init 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient: _add_conns ranks=[0,1,8,4,5,3,2,7,6] 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient(hunting): picked mon.noname-a con 0x7f61980a3e30 addr v1:172.21.15.121:6789/0 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient(hunting): picked mon.noname-f con 0x7f61980a4210 addr v1:172.21.15.153:6789/0 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient(hunting): picked mon.noname-e con 0x7f619813d050 addr v1:172.21.15.121:6793/0 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61980737f0 con 0x7f61980a3e30 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61980734a0 con 0x7f61980a4210 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 1 -- --> v1:172.21.15.121:6793/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61980a5b60 con 0x7f619813d050 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:26.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619f4d4640 10 monclient(hunting): authenticate will time out at 886.174737s 2024-09-13T13:53:26.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619d270640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f61980a3e30 0x7f6198073fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:46680/0 (socket says 172.21.15.121:46680) 2024-09-13T13:53:26.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619d270640 1 -- 172.21.15.121:0/60266485 learned_addr learned my addr 172.21.15.121:0/60266485 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:26.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.545+0000 7f619da71640 1 --1- 172.21.15.121:0/60266485 >> v1:172.21.15.121:6793/0 conn(0x7f619813d050 0x7f619813f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6793/0 says I am v1:172.21.15.121:42924/0 (socket says 172.21.15.121:42924) 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.547+0000 7f619d270640 1 -- 172.21.15.121:0/60266485 >> v1:172.21.15.121:6789/0 conn(0x7f61980a3e30 legacy=0x7f6198073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f619da71640 1 -- 172.21.15.121:0/60266485 >> v1:172.21.15.121:6793/0 conn(0x7f619813d050 legacy=0x7f619813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 <== mon.0 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610105898 0 0) 0x7f61980737f0 con 0x7f61980a3e30 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): my global_id is 4770 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6184003520 con 0x7f61980a3e30 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 <== mon.8 v1:172.21.15.121:6793/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221014072 0 0) 0x7f61980a5b60 con 0x7f619813d050 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient(hunting): my global_id is 4751 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 --> v1:172.21.15.121:6793/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61980737f0 con 0x7f619813d050 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f619ca6f640 1 -- 172.21.15.121:0/60266485 >> v1:172.21.15.153:6789/0 conn(0x7f61980a4210 legacy=0x7f61980a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 <== mon.8 v1:172.21.15.121:6793/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3522962913 0 0) 0x7f61980737f0 con 0x7f619813d050 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 >> v1:172.21.15.153:6789/0 conn(0x7f61980a4210 legacy=0x7f61980a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 >> v1:172.21.15.121:6789/0 conn(0x7f61980a3e30 legacy=0x7f6198073fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: _finish_hunting 0 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 monclient: found mon.noname-e 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: _send_mon_message to mon.noname-e at v1:172.21.15.121:6793/0 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 --> v1:172.21.15.121:6793/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6198143b90 con 0x7f619813d050 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: _finish_auth 0 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: _check_auth_tickets 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 <== mon.8 v1:172.21.15.121:6793/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6190001dc0 con 0x7f619813d050 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f619f4d4640 5 monclient: authenticate success, global_id 4751 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-13T13:53:26.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: dump: 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.561 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f619f4d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 <== mon.8 v1:172.21.15.121:6793/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6190002090 con 0x7f619813d050 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 1 -- 172.21.15.121:0/60266485 <== mon.8 v1:172.21.15.121:6793/0 5 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6190004ea0 con 0x7f619813d050 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f619f4d4640 10 monclient: get_monmap_and_config success 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f619f4d4640 4 set_mon_vals no callback set 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.548+0000 7f618bfff640 10 monclient: dump: 2024-09-13T13:53:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 10 monclient: shutdown 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 1 -- 172.21.15.121:0/60266485 >> v1:172.21.15.121:6793/0 conn(0x7f619813d050 legacy=0x7f619813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 1 -- 172.21.15.121:0/60266485 shutdown_connections 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 1 -- 172.21.15.121:0/60266485 >> 172.21.15.121:0/60266485 conn(0x7f61980a9430 msgr2=0x7f61980a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 1 -- 172.21.15.121:0/60266485 shutdown_connections 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 1 -- 172.21.15.121:0/60266485 wait complete. 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 10 monclient: build_initial_monmap 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.549+0000 7f619f4d4640 10 monclient: monmap: 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 0 2024-09-13T13:53:26.564 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 0.000000 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:created 0.000000 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 0 (unknown) 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 1 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.noname-a 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.noname-f 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.noname-b 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.noname-g 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.noname-c 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.noname-h 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.noname-d 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.noname-i 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.noname-e 2024-09-13T13:53:26.565 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 1 Processor -- start 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 1 -- start start 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient: init 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient: _reopen_session rank -1 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient: _add_conns ranks=[4,5,2,1,8,3,0,6,7] 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient(hunting): picked mon.noname-c con 0x7f61980a3e30 addr v1:172.21.15.121:6791/0 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient(hunting): picked mon.noname-h con 0x7f61980a4210 addr v1:172.21.15.153:6791/0 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient(hunting): picked mon.noname-b con 0x7f619813d050 addr v1:172.21.15.121:6790/0 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 1 -- --> v1:172.21.15.121:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6198120300 con 0x7f619813d050 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 1 -- --> v1:172.21.15.121:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f619809e150 con 0x7f61980a3e30 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 1 -- --> v1:172.21.15.153:6791/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f619809fee0 con 0x7f61980a4210 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient(hunting): _renew_subs 2024-09-13T13:53:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619f4d4640 10 monclient(hunting): authenticate will time out at 886.179620s 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619d270640 1 --1- >> v1:172.21.15.121:6791/0 conn(0x7f61980a3e30 0x7f619809d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.121:6791/0 says I am v1:172.21.15.121:33478/0 (socket says 172.21.15.121:33478) 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.550+0000 7f619d270640 1 -- 172.21.15.121:0/4280188209 learned_addr learned my addr 172.21.15.121:0/4280188209 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f619d270640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.121:6791/0 conn(0x7f61980a3e30 legacy=0x7f619809d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f619da71640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.121:6790/0 conn(0x7f619813d050 legacy=0x7f61981d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543665722 0 0) 0x7f619809e150 con 0x7f61980a3e30 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): my global_id is 4684 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f617c003520 con 0x7f61980a3e30 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.2 v1:172.21.15.121:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591408991 0 0) 0x7f6198120300 con 0x7f619813d050 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 10 monclient(hunting): my global_id is 4736 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.552+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f619809e150 con 0x7f619813d050 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619ca6f640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.153:6791/0 conn(0x7f61980a4210 legacy=0x7f619809da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4010068281 0 0) 0x7f617c003520 con 0x7f61980a3e30 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6198120300 con 0x7f61980a3e30 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f618c0047e0 con 0x7f61980a3e30 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.568 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): _finish_auth 0 2024-09-13T13:53:26.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.5 v1:172.21.15.153:6791/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367506454 0 0) 0x7f619809fee0 con 0x7f61980a4210 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): _negotiate 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): _init_auth method 2 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): my global_id is 4721 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.153:6791/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f617c003520 con 0x7f61980a4210 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.2 v1:172.21.15.121:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1564446475 0 0) 0x7f619809e150 con 0x7f619813d050 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f619809fee0 con 0x7f619813d050 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.2 v1:172.21.15.121:6790/0 3 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f6190001e20 con 0x7f619813d050 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient(hunting): dump: 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.570 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.571 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3267237 0 0) 0x7f6198120300 con 0x7f61980a3e30 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.153:6791/0 conn(0x7f61980a4210 legacy=0x7f619809da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.121:6790/0 conn(0x7f619813d050 legacy=0x7f61981d3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: _finish_hunting 0 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 monclient: found mon.c 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6198120020 con 0x7f61980a3e30 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: _finish_auth 0 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: _check_auth_tickets 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 5 monclient: authenticate success, global_id 4684 2024-09-13T13:53:26.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 10 monclient: _renew_subs 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f619809f7c0 con 0x7f61980a3e30 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f618c004ce0 con 0x7f61980a3e30 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: handle_config config(0 keys) 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 6 ==== mon_map magic: 0 ==== 722+0+0 (unknown 2256400014 0 0) 0x7f618c005250 con 0x7f61980a3e30 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f6189ffb640 10 monclient: dump: 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:epoch 1 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:fsid 21ebbdf6-8985-4682-9234-cdd2d2c1c2db 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:last_changed 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:created 2024-09-13T13:50:47.391384+0000 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:min_mon_release 19 (squid) 2024-09-13T13:53:26.573 INFO:teuthology.orchestra.run.smithi121.stderr:election_strategy: 3 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:0: v1:172.21.15.121:6789/0 mon.a 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:1: v1:172.21.15.153:6789/0 mon.f 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:2: v1:172.21.15.121:6790/0 mon.b 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:3: v1:172.21.15.153:6790/0 mon.g 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:4: v1:172.21.15.121:6791/0 mon.c 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:5: v1:172.21.15.153:6791/0 mon.h 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:6: v1:172.21.15.121:6792/0 mon.d 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:7: v1:172.21.15.153:6792/0 mon.i 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:8: v1:172.21.15.121:6793/0 mon.e 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f618b7fe640 4 set_mon_vals no callback set 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 10 monclient: _renew_subs 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:26.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.553+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- mon_subscribe({osdmap=0}) -- 0x7f61981d5210 con 0x7f61980a3e30 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 7 ==== mgrmap(e 5) ==== 50384+0+0 (unknown 799781749 0 0) 0x7f618c0053f0 con 0x7f61980a3e30 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f619f4d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f619f4d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f619f4d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 8 ==== osd_map(27..27 src has 1..27) ==== 3506+0+0 (unknown 992214784 0 0) 0x7f618c04e280 con 0x7f61980a3e30 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6198077320 con 0x7f61980a3e30 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.554+0000 7f619ca6f640 10 monclient: get_auth_request con 0x7f617c040250 auth_method 0 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.559+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (unknown 1092875540 0 1080655425) 0x7f618c0178b0 con 0x7f61980a3e30 2024-09-13T13:53:26.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.559+0000 7f6189ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-13T13:53:26.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.559+0000 7f6189ffb640 10 monclient: _finish_command 1 = system:0 2024-09-13T13:53:26.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.736+0000 7f619f4d4640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T13:53:26.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.736+0000 7f619f4d4640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T13:53:26.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.736+0000 7f619f4d4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.121:6791/0 2024-09-13T13:53:26.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.736+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 --> v1:172.21.15.121:6791/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f619813fb50 con 0x7f61980a3e30 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f618c016d20 con 0x7f61980a3e30 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f6189ffb640 10 monclient: handle_config config(1 keys) 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f618b7fe640 4 set_mon_vals no callback set 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f618b7fe640 10 set_mon_vals mon_health_to_clog = false 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f6189ffb640 1 -- 172.21.15.121:0/4280188209 <== mon.4 v1:172.21.15.121:6791/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (unknown 4099060384 0 0) 0x7f618c0171b0 con 0x7f61980a3e30 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f6189ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-13T13:53:26.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.746+0000 7f6189ffb640 10 monclient: _finish_command 2 = system:0 2024-09-13T13:53:26.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.153:6812/1206688214 conn(0x7f617c040250 legacy=0x7f617c042710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 10 monclient: shutdown 2024-09-13T13:53:26.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-13T13:53:26.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 >> v1:172.21.15.121:6791/0 conn(0x7f61980a3e30 legacy=0x7f619809d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T13:53:26.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 shutdown_connections 2024-09-13T13:53:26.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 >> 172.21.15.121:0/4280188209 conn(0x7f61980a9430 msgr2=0x7f61980a88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T13:53:26.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.748+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 shutdown_connections 2024-09-13T13:53:26.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-09-13T13:53:26.749+0000 7f619f4d4640 1 -- 172.21.15.121:0/4280188209 wait complete. 2024-09-13T13:53:26.760 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T13:53:26.760 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T13:53:26.760 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T13:53:26.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:26.792 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T13:53:26.793 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T13:53:26.793 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:26.856 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T13:53:26.856 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T13:53:26.856 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:26.926 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T13:53:26.926 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T13:53:26.926 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:26.958 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T13:53:26.958 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T13:53:26.958 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.039 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T13:53:27.039 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T13:53:27.039 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.105 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T13:53:27.105 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T13:53:27.105 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T13:53:27.105 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.173 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T13:53:27.173 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T13:53:27.173 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T13:53:27.173 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.185 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T13:53:27.185 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T13:53:27.185 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.237 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T13:53:27.237 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T13:53:27.237 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.289 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T13:53:27.289 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-13T13:53:27.289 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.340 INFO:tasks.ceph.mon.d:Stopped 2024-09-13T13:53:27.340 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-09-13T13:53:27.340 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.390 INFO:tasks.ceph.mon.e:Stopped 2024-09-13T13:53:27.391 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-09-13T13:53:27.391 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.403 INFO:tasks.ceph.mon.f:Stopped 2024-09-13T13:53:27.403 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-13T13:53:27.404 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.457 INFO:tasks.ceph.mon.g:Stopped 2024-09-13T13:53:27.457 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-13T13:53:27.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.509 INFO:tasks.ceph.mon.h:Stopped 2024-09-13T13:53:27.510 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-13T13:53:27.510 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T13:53:27.562 INFO:tasks.ceph.mon.i:Stopped 2024-09-13T13:53:27.562 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T13:53:27.562 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-13T13:53:27.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:53:27.591 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T13:53:27.809 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:53:27.809 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T13:53:27.898 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:53:27.899 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T13:53:27.949 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:53:27.949 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T13:53:28.094 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:53:28.094 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T13:53:28.167 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:53:28.167 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T13:53:28.223 INFO:tasks.ceph:Archiving mon data... 2024-09-13T13:53:28.224 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.a.tgz 2024-09-13T13:53:28.225 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-13T13:53:28.240 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.NyMHj63BAg 2024-09-13T13:53:28.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NyMHj63BAg 2024-09-13T13:53:28.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.NyMHj63BAg 2024-09-13T13:53:28.437 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.NyMHj63BAg is 228KB 2024-09-13T13:53:28.531 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.NyMHj63BAg 2024-09-13T13:53:28.546 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.b.tgz 2024-09-13T13:53:28.547 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-13T13:53:28.600 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.sotRz2oi70 2024-09-13T13:53:28.601 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.sotRz2oi70 2024-09-13T13:53:28.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.sotRz2oi70 2024-09-13T13:53:28.784 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.sotRz2oi70 is 228KB 2024-09-13T13:53:28.877 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.sotRz2oi70 2024-09-13T13:53:28.892 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.c.tgz 2024-09-13T13:53:28.892 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-13T13:53:28.946 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.3s7Or6upAZ 2024-09-13T13:53:28.947 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3s7Or6upAZ 2024-09-13T13:53:29.048 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.3s7Or6upAZ 2024-09-13T13:53:29.131 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.3s7Or6upAZ is 228KB 2024-09-13T13:53:29.224 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.3s7Or6upAZ 2024-09-13T13:53:29.239 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.d.tgz 2024-09-13T13:53:29.239 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-13T13:53:29.293 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.SE8Yo98zz6 2024-09-13T13:53:29.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.SE8Yo98zz6 2024-09-13T13:53:29.395 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.SE8Yo98zz6 2024-09-13T13:53:29.480 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.SE8Yo98zz6 is 228KB 2024-09-13T13:53:29.577 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.SE8Yo98zz6 2024-09-13T13:53:29.592 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.e.tgz 2024-09-13T13:53:29.592 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-09-13T13:53:29.647 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.kJRuzZqCED 2024-09-13T13:53:29.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.kJRuzZqCED 2024-09-13T13:53:29.748 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.kJRuzZqCED 2024-09-13T13:53:29.830 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.kJRuzZqCED is 228KB 2024-09-13T13:53:29.923 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.kJRuzZqCED 2024-09-13T13:53:29.938 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.f.tgz 2024-09-13T13:53:29.938 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-13T13:53:29.957 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.SSOf4ozhid 2024-09-13T13:53:29.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.SSOf4ozhid 2024-09-13T13:53:30.152 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.SSOf4ozhid 2024-09-13T13:53:30.238 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.SSOf4ozhid is 228KB 2024-09-13T13:53:30.333 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.SSOf4ozhid 2024-09-13T13:53:30.349 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.g.tgz 2024-09-13T13:53:30.349 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-13T13:53:30.405 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.qq4F1fBixU 2024-09-13T13:53:30.405 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.qq4F1fBixU 2024-09-13T13:53:30.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.qq4F1fBixU 2024-09-13T13:53:30.591 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.qq4F1fBixU is 228KB 2024-09-13T13:53:30.687 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.qq4F1fBixU 2024-09-13T13:53:30.702 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.h.tgz 2024-09-13T13:53:30.702 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-13T13:53:30.757 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.O8CvxViYAt 2024-09-13T13:53:30.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.O8CvxViYAt 2024-09-13T13:53:30.858 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.O8CvxViYAt 2024-09-13T13:53:30.942 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.O8CvxViYAt is 228KB 2024-09-13T13:53:31.035 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.O8CvxViYAt 2024-09-13T13:53:31.050 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/data/mon.i.tgz 2024-09-13T13:53:31.052 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-13T13:53:31.105 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.8o45z9tPYj 2024-09-13T13:53:31.105 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.8o45z9tPYj 2024-09-13T13:53:31.206 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.8o45z9tPYj 2024-09-13T13:53:31.289 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.8o45z9tPYj is 228KB 2024-09-13T13:53:31.383 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.8o45z9tPYj 2024-09-13T13:53:31.398 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T13:53:31.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T13:53:31.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T13:53:31.486 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T13:53:31.521 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-13T13:53:31.521 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T13:53:31.522 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/remote/smithi121/crash 2024-09-13T13:53:31.523 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T13:53:31.554 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/remote/smithi153/crash 2024-09-13T13:53:31.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T13:53:31.585 INFO:tasks.ceph:Compressing logs... 2024-09-13T13:53:31.585 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:53:31.598 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-13T13:53:31.623 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-13T13:53:31.623 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35219.log 2024-09-13T13:53:31.623 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36213.log 2024-09-13T13:53:31.623 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37982.log 2024-09-13T13:53:31.624 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36766.log 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35219.log.gz 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36171.log 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37982.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36213.log.gz 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36598.log 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37982.log.gz 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36255.log 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36766.log.gz 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36045.log 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36598.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36171.log.gz 2024-09-13T13:53:31.645 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36598.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36045.log: gzip -5 --verbose -- 89.7% /var/log/ceph/ceph-client.admin.36724.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36255.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36430.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36045.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36724.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36430.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36724.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31791.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36430.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.31791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36339.log 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37648.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31791.log.gz 2024-09-13T13:53:31.646 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35218.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36339.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gzgzip 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36850.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35471.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35218.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36339.log.gz 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35791.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35218.log.gz 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35471.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36850.log.gz 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36003.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36472.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35791.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35471.log.gz 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37522.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35791.log.gz 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35470.log 2024-09-13T13:53:31.647 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36472.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36003.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35468.log 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36472.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35470.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37522.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37774.log 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35468.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35470.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35468.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37270.log 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37228.log 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37270.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37774.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32204.log 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37270.log.gz 2024-09-13T13:53:31.648 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36556.log 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35035.log 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37228.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36556.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32204.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36892.log 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35035.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36556.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr: -- /var/log/ceph/ceph-client.admin.32135.log 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35035.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37312.log 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.32135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34875.log 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr: 96.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36892.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34993.log 87.8% 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32135.log.gz 2024-09-13T13:53:31.649 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34875.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37312.log.gz 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36297.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34875.log.gz 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37438.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34993.log.gz 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34662.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36297.log: /var/log/ceph/ceph-client.admin.37438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35833.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34662.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36297.log.gz 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37438.log.gz 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37480.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35833.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34662.log.gz 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37186.log 2024-09-13T13:53:31.650 INFO:teuthology.orchestra.run.smithi121.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35833.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37480.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35467.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37396.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37186.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35467.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37144.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36682.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37396.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35467.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37354.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37396.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36682.log: gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37144.log.gz 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr: /var/log/ceph/ceph-mon.a.log 2024-09-13T13:53:31.651 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37354.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36682.log.gz 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37858.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37354.log.gz 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35215.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35215.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37858.log.gz 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr: /var/log/ceph/ceph-mon.e.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36976.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35215.log.gz 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37732.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.e.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36388.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37732.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35623.log 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37732.log.gz 2024-09-13T13:53:31.652 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36388.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36934.log 2024-09-13T13:53:31.653 INFO:teuthology.orchestra.run.smithi121.stderr: 87.8%/var/log/ceph/ceph-client.admin.35623.log: -- replaced with /var/log/ceph/ceph-client.admin.36388.log.gz 2024-09-13T13:53:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36808.log 2024-09-13T13:53:31.653 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35623.log.gz 2024-09-13T13:53:31.653 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36934.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27132.log 2024-09-13T13:53:31.668 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36808.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36934.log.gz 2024-09-13T13:53:31.668 INFO:teuthology.orchestra.run.smithi121.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36808.log.gz 2024-09-13T13:53:31.668 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-09-13T13:53:31.668 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35917.log 2024-09-13T13:53:31.668 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.tmp-client.admin.27132.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27132.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35917.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35917.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37942.log 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr: 88.4%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35469.log 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37942.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37942.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35211.log 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35469.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35469.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36129.log 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35211.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35211.log.gz 2024-09-13T13:53:31.669 INFO:teuthology.orchestra.run.smithi121.stderr: 95.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35959.log 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37060.log 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36129.log.gz 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37606.log 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37060.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35959.log.gz 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34900.log 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37060.log.gz 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35707.log 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34900.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37606.log.gz 2024-09-13T13:53:31.670 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-09-13T13:53:31.671 INFO:teuthology.orchestra.run.smithi121.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34900.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35665.log 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.d.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35707.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35665.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35665.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35875.log 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36640.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37816.log 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35875.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35875.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37102.log 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37816.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37816.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37102.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37102.log.gz 2024-09-13T13:53:31.672 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35214.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34982.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32077.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34982.log: /var/log/ceph/ceph-mon.g.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34983.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.4.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32077.log.gz 2024-09-13T13:53:31.677 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34948.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32117.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34947.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31906.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34948.log.gz 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34947.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32117.log.gz 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31906.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.34983.log.gz 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31906.log.gz 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32037.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.h.log: /var/log/ceph/ceph-mon.i.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31989.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr: 96.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.32037.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-13T13:53:31.678 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32037.log.gz 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph.log 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.31989.log: gzip -5 --verbose 90.8% -- replaced with /var/log/ceph/ceph-client.admin.34947.log.gz 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr: -- /var/log/ceph/ceph-client.admin.34981.log 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34981.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31989.log.gz 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31946.log 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mgr.x.log: 94.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.34981.log.gz 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31946.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-13T13:53:31.679 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.f.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31946.log.gz 2024-09-13T13:53:31.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36514.log 2024-09-13T13:53:31.680 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35214.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35214.log.gz 2024-09-13T13:53:31.684 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35077.log 2024-09-13T13:53:31.685 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.36514.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36514.log.gz 2024-09-13T13:53:31.710 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.3.log: 91.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-13T13:53:31.711 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35749.log 2024-09-13T13:53:31.711 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35077.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35077.log.gz 2024-09-13T13:53:31.721 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37564.log 2024-09-13T13:53:31.721 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35749.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35749.log.gz 2024-09-13T13:53:31.729 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37018.log 2024-09-13T13:53:31.738 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-09-13T13:53:31.739 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35456.log 2024-09-13T13:53:31.743 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37018.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37564.log.gz 2024-09-13T13:53:31.744 INFO:teuthology.orchestra.run.smithi153.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-09-13T13:53:31.746 INFO:teuthology.orchestra.run.smithi121.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-13T13:53:31.746 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37900.log 2024-09-13T13:53:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.35456.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35456.log.gz 2024-09-13T13:53:31.747 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34951.log 2024-09-13T13:53:31.748 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.37900.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37900.log.gz 2024-09-13T13:53:31.748 INFO:teuthology.orchestra.run.smithi121.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-09-13T13:53:31.749 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.34951.log: 89.6% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34951.log.gz 2024-09-13T13:53:31.749 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37018.log.gz 2024-09-13T13:53:31.749 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-09-13T13:53:31.755 INFO:teuthology.orchestra.run.smithi121.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-13T13:53:31.758 INFO:teuthology.orchestra.run.smithi121.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-13T13:53:31.781 INFO:teuthology.orchestra.run.smithi153.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-09-13T13:53:31.788 INFO:teuthology.orchestra.run.smithi121.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-13T13:53:31.798 INFO:teuthology.orchestra.run.smithi121.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-09-13T13:53:31.821 INFO:teuthology.orchestra.run.smithi153.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-13T13:53:31.825 INFO:teuthology.orchestra.run.smithi121.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-13T13:53:32.176 INFO:teuthology.orchestra.run.smithi153.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-13T13:53:32.176 INFO:teuthology.orchestra.run.smithi121.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-13T13:53:32.178 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-09-13T13:53:32.178 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m0.567s 2024-09-13T13:53:32.178 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m1.466s 2024-09-13T13:53:32.179 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.119s 2024-09-13T13:53:32.179 INFO:teuthology.orchestra.run.smithi153.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-13T13:53:32.181 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-13T13:53:32.181 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m0.537s 2024-09-13T13:53:32.181 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m1.587s 2024-09-13T13:53:32.181 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.086s 2024-09-13T13:53:32.181 INFO:tasks.ceph:Archiving logs... 2024-09-13T13:53:32.182 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/remote/smithi121/log 2024-09-13T13:53:32.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:53:32.508 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/remote/smithi153/log 2024-09-13T13:53:32.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T13:53:32.772 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T13:53:32.782 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T13:53:32.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:53:32.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T13:53:32.878 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T13:53:32.886 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T13:53:32.887 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:53:32.889 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:53:32.901 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:53:32.904 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-09-13T13:53:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:53:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-09-13T13:53:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 39 -41us[ -41us] +/- 36ms 2024-09-13T13:53:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -474us[ -493us] +/- 66ms 2024-09-13T13:53:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +556us[ +556us] +/- 67ms 2024-09-13T13:53:32.906 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:53:32.908 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T13:53:32.908 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-09-13T13:53:32.908 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 40 -37us[ -28us] +/- 36ms 2024-09-13T13:53:32.908 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 39 -476us[ -467us] +/- 66ms 2024-09-13T13:53:32.908 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv03.front.sepia.ceph.com 3 6 377 39 +563us[ +573us] +/- 67ms 2024-09-13T13:53:32.909 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T13:53:32.909 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T13:53:32.919 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T13:53:32.919 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T13:53:32.928 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:53:33.009 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T13:53:33.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:53:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:53:33.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T13:53:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:53:33.125 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T13:53:33.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T13:53:33.143 INFO:teuthology.task.internal:Duration was 528.499774 seconds 2024-09-13T13:53:33.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T13:53:33.152 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T13:53:33.152 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:53:33.155 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T13:53:33.191 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:53:33.205 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T13:53:33.652 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T13:53:33.652 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-09-13T13:53:33.652 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:53:33.673 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-09-13T13:53:33.673 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T13:53:33.718 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T13:53:33.718 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:53:33.721 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T13:53:33.785 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T13:53:33.785 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:53:33.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T13:53:33.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T13:53:33.937 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T13:53:33.937 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:53:33.962 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T13:53:33.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T13:53:33.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:53:34.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:53:34.027 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-09-13T13:53:34.051 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-09-13T13:53:34.080 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:53:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:53:34.105 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:53:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:53:34.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T13:53:34.146 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T13:53:34.147 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/remote/smithi121 2024-09-13T13:53:34.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:53:34.180 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903517/remote/smithi153 2024-09-13T13:53:34.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T13:53:34.214 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T13:53:34.214 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:53:34.217 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T13:53:34.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T13:53:34.275 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T13:53:34.275 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T13:53:34.360 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T13:53:34.360 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:53:34.363 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T13:53:34.375 INFO:teuthology.orchestra.run.smithi121.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:53 /home/ubuntu/cephtest 2024-09-13T13:53:34.378 INFO:teuthology.orchestra.run.smithi153.stdout: 265602 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 13:53 /home/ubuntu/cephtest 2024-09-13T13:53:34.379 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T13:53:34.389 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T13:53:34.412 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/many workloads/rados_5925} duration: 528.499773979187 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T13:53:34.412 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:53:34.502 INFO:teuthology.run:pass